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Foreword 


This research paper forms part of Cedefop’s Comparing VET qualifications project 
and contributes to the development of methodologies for the analysis and 
comparison of the content and profile of vocational education and training 
programmes and qualifications. 

While much effort traditionally has been put into the analysis and comparison 
of VET institutions and structures, systematic studies on how VET qualifications 
differ in terms of knowledge, skills and competence are less developed. The 
Comparing VET qualifications project (') addresses this imbalance and seeks to 
establish a methodological basis on which future work in this area can build. 

A wide range of stakeholders, including policy-makers, education and training 
practitioners and researchers, can benefit from robust and_ scalable 
methodologies, allowing them review and renew more systematically the content 
of their programmes and qualifications. And, as illustrated by the current research 
paper, comparative methodologies are relevant for a wide range of stakeholders 
operating at different levels, at local and regional, as well as at national and 
international levels. 

The project starts from the increasing use of learning outcomes in countries 
across Europe and the world (Cedefop, 2009; 2017; 2020). The use of learning 
outcomes for defining and describing qualifications allows for systematic analysis 
and comparison of what countries expect their VET candidates to know, be able to 
do and understand. The current research paper elaborates on how this analysis 
and comparison of learning outcomes can support the review and renewal of 
qualifications by improving the dialogue (feedback loop) between VET providers 
and the labour market. By exploring how systematically to gather, analyse, 
compare and share information on the demand for, and supply of, skills and 
competences, at international as well as national level, the paper contributes 
directly to the long-term development of robust and scalable methods in this area. 


Jurgen Siebel Loukas Zahilas 
Executive Director Head of Department for VET 
systems and Institutions 


(') Between 2015 and 2017, Cedefop carried out a pilot study comparing 10 VET qualifications in 
10 European countries. The purpose of this study was to gain insight into the similarities and 
differences between countries regarding the content and profile of their qualifications. The 
methodology developed for this study was further tested in cooperation with the ETF and 
UNESCO, where four of the original 10 qualifications were compared in 26 countries worldwide. 
The lessons from this pilot led to the launching of the Comparing VET qualifications project. 


Acknowledgements 


This research paper was produced by Cedefop, department for systems and 
institutions under the supervision of Loukas Zahilas. Cedefop expert Jens 
Bjornavold initiated and was responsible for the publication and the research 
conducted under the project Comparing vocational education and _ training 
qualifications: towards a European comparative methodology. A consortium 
consisting of 3s Unternehmensberatung GmbH and Ockham IPS, led by Karin 
Luomi-Messerer, was responsible for the overall research carried out under this 
project. Simon Broek, Ockham IPS, was responsible for the coordination and 
preparation of the current research paper. 


Contents 


FOLOWONGE fonc afeccstete seek ce camemadonte ccs Addsastescnstasteawenmdqnactdans Mlactstecessecptedebacnends 1 
COMI TS isa case tsineaassavseatiadinunsanneiwardaais iecteneste wipes tandeeadunied gnvadaste viuatadiuwaascn 3 
EX@Cutive SUMMANY cesecstdecancestbessents ea tedea vena aad dedetietabvashaweesaeneienesdanessans 7 
1. Background of the study, key objectives and methodological 
GO PFOACH seine ei ececeades eh ee me code cdode ceathls aie eos eats beweauieccande dances 18 
1.1. Setting the scene: relevance of qualifications and 
TeC@OUDACK IOOPS: sassk ca einctssiauilelsned etilseu lhe haagews teed ianey 18 
1.1.1. _ Feedback loop between VET and the labour 
market and VET governance models .................... 19 
1.1.2. Learning outcomes, continuous dialogue and 
necessary data to complete the feedback loop ..... 24 
1.2. Key objectives and main research questions ..............:0006 27 
1.3. | Methodological approach and overview ................:cccccccee 28 
1.3.1. Methodological approach .............eeeeeeeeeeeeeeeeeeeteees 28 
TeGe2i, SQVEIVIOWE, 5 Nase Gatinisachonntnste a enaeeaaeee 33 
2. Exploration of methods for completing the feedback loop .............. 34 
as Pn | 011010 [0 (61 | (0) 9 ee ee 34 
2.2. Skill mismatch analyses and the European skills and jobs 
SUIVOY 2?eve: ceca cadetesctiirs feacigetelicen fede Geetvandeatas ae aeenatetieee tease 35 
2.3. | Vacancy analyses and Cedefop’s skills online vacancy 
analysis tool for EUrOpe ...........:ccccccccessseeeeccceeeeeeseeeeeereeeeeeeees 37 
2.4. Forecasting approaches and Cedefop’s skills forecast ....... 39 
2.5. VET graduate tracking MEASUIES ...........ccceeeeeeeeeeeeeeeeeeeeees 42 
2.6. Employer reflection SurveyS ..............:::::ccccceeeeeeeeseeeeneeeeeeeees 44 
2.6.1. | Surveys that address employer perception of 
and demand for qualifications.............eeeeeeeeeeeees 45 
2.6.2. Employer reflection surveys without reference to 
learning outcomes in qualifications... 47 
2.6.3. Employer reflection surveys with reference to 
learning outcomes included in qualifications ......... 48 
2.7. Closing the learning outcomes feedback loop: an 
assessment of different approaches... eeeeeeeeeeeeeees 49 
3. Employer reflection on achieved learning outcomes: surveys 
CXAMINGd: teen eae Miedo eae ei eitene en eas 52 
x Fas a [0100 [0 (61 | (0) 9 ee 52 


Review and renewal of qualifications: 
towards methodologies for analysing and comparing learning outcomes 


3.2. Types of organisation involved .............:::::ccceeseeeeeeeeeeeeeetees 55 
3.3. | Main objective of an employer reflection survey............. 57 
3.4. | Survey scope: geography, education level sectors, 
OCCUPATIONS /QUALITICATIONS os sis ccetscdecenerevacd oudvelereoneadectenonansenes 58 
3.5. Targeted respondent .............ccccccecessseeeeeeeeeeeeeeseeeeeeeeeeeeees 61 
3.6. General SUrVEY APPFOACN 0.0.0... eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeesenneeees 64 
3.7. Content of the questionnaires. ............. cc eeceeceeeeeeeeeeeeeeeeeees 68 


3.8. | Challenges associated with employer reflection surveys .... 77 


3.9. | Relevance and usefulness of the employer 
Petlection-SUVEY. veka ake ae ees nee eee cade ht 80 


4. Prototype of a survey for employer reflection on (VET) graduates . 83 


4.1. Prototype general outline... eeeeseeeeeeeceeeeeeeeeeseeeeeees 83 
4.2. Methodological approach to the pre-test... ees 84 
4.3. Askills typology for an employer reflection survey.............. 86 
4.3.1. Use of international reference points................06 86 
4.3.2. _ Use of existing national skills typologies................ 90 
4.3.3. | Conceptual basis for the ERS skills 
TETELGMCS POU sac rric store eeas serena estiectotdeedieneatisuss 93 
4.3.4. Skills reference point for the ERS pre-test............. 98 
4.4. Questionnaires for VET providers, graduates and 
CMplOVErS/SUDSTVISONS vaceciey condi ceceecsansetet teedv ewe ote mess 100 
4.5. Pre-test approach and results 20.0.0... eeeeeeeeeeeeeeeeeeeeeeeneeee 104 
4.5.1. Implementation and responses ............eee eee 104 
4.5.2. Results for a single qualification (Netherlands 
ROC Tilburg: healthcare assistant) ..........:08 106 
4.5.3. | Comparing qualifications: overall analysis results 112 
4.6... EMGENQINGISSUCS 54.2: henner: stead tcntdtandndetictennedantienteaenaes 118 
5. Conclusions and recommendations ............::cccccceeeeeeeeeeeeeteeeeeeees 120 
Bide 4 “GONEIUSIONS os saxjsansuitacn cies ted axiandesah tase Rim ia cat aden nes 120 
D2. - “RECOMIMENGALIONS sccicaidets decanter te stvchenl estee dune eto.dotareects 126 
PRGCMOMY NS ce sises see nce durante eSoset dae cbeeta tastes paedenedeyenvarassvoneedestntash aiestvenesess nests 131 
REPEFENCES.............cccccceecececeeeceeeeeeeceeeeeeeceaeceneceaeceaecaeeceaeceaeeeaeeeeeseeeeseeeees 134 
FUMth@n: FOAGING) aiczecncttietivecestsacetaves ental vanatetzces tans Setecdedzesacaeteeaeelealadan 139 
Further-Sources Of InfOrMatlonncciicccacesstekieteatanivolensncetesieresevihieienees 140 


Contents 


Tables, figures and boxes 


Tables 
1 Template for desk research at country level.............c:ceeeeeeeeeeeeenteeees 31 
2. Countries and qualification profiles ............:cccceeeeeeeeeeeeeeeeeeeeeeeeeneeeees 33 
3. Overview of questionnaire StrUCtUIe ............ccccceeeceeseteeeeeeeeeeeseeenteaees 36 
4 Overview of initiators and coordinators/conductors for the six 

OXAMPIOS SCIOCIO” oe. icccccssssencccccenesssesteacoetsentsaseatasconsnenentecetsccensaenacaece 56 
5. Main objectives of ERS, Dy COUNIIY sacciscecscciccesonicndsstcerseeasricnnicnnrtees 57 
6. Overview of economic sectors identified in ERS examples.............. 59 
7. Occupations specified in ERS (LT 2, AT), by corresponding 

ECONOMIC SOCTHOM...........cccccceecceecceecceeccececececeeccecceeceeeccencceeceeeeesneseneetess 60 
8. Overview of target respondent demarcations and descriptions 

identified in the ERS examples ..............:cccceescsceeeeesseeeeeeesseeeeteeeaneeees 62 
9. Overview of typologies identified in ERS examples................::000 71 
10. ESS graduate attributes scale (GAS-E)............ecccceeeesseeeeeeseeeeeeeeeeees 74 


11. Overview of questions and scales/answer categories related to 
assessing skills and competences (broadly linked to learning 


GUNCOIIGS) cxgtesiant cee nerit a ccres doe tteen eeieadecoeeceaeweletcasieteeeanieee een onena 76 
12. Draft ESCO skills and knowledge classification, 

Pevised 4 OGioDer 2019 scisisaisscaisatvesdsanana mecdanenloapasndnninsiasanminnaaiianiats 87 
13. Describing relationships and objects .............:::ccccceeeeseseeceeeeeeeeeeeeeees 95 
14. Questionnaire for VET providers ..............c:cceeeeeeeeeeeeeneeeeeessaeeeeeneaes 101 
15. Questionnaire for VET graduates .....0.......cceeeeeeeeeeeseeeeeesseeeeeenaes 102 
16. Questionnaire for EMplOyerS.............cceeeeeeeeeeeeeeeeeeeeeneeeeeeeeteeeeeeeees 103 
17. Respondents in the ERS pre-test ..............:ceeeeeeeeeeseeeeeeesseeeeteeaes 105 
18. Most and least important learning outcomes 

(Netherlands healthcare assistant) ............eecceeeeeeeeeeeeeeeentneeeeeeeeeeeee 111 
19. Most and least important learning outcomes all 

qualification profiles ANd COUNTIICS........... ccc cceeeeeeeeeeeeeeeeeeeeeeeeeeeeenaes 117 
20. ReS@arch tai ........ccecccecceecececeeeeeeeceeeeeeeeeeeeeeeeeeeeeeseeeeeeeeeeeeeeeeeeeeeees 144 
Figures 
1. Positioning the approaches and tools analysed for closing the 

learning outcomes feedback lOOp ..............::cseccceeeeeeeeeeeeeeeeeeeeeeeeneeeees 11 
2. Opportunities and objectives of an ERS approach..................:cee 12 
3. Basic model of feedback mechanism between VET and labour 

Mar Kel riaieetiieudeuenlici hei ives ell csaen achieved aie chiens 20 
4. Four types of formal feedback mechanism between VET and labour 

IMMAIIKGT igh dsc v dis caasdeastseinesvaiusychvancaiseved iageasebaneoaasus¥endazsddavadaindeascscynensae 21 


Review and renewal of qualifications: 
towards methodologies for analysing and comparing learning outcomes 


5. Learning outcomes feedback lOOP............:::::ccccecesssseeeceeeeeeeeeeeeneeeees 25 
6. — Analytical framework and methodological steps ............: cee 29 
7. Most requested skills in online job vacancies for 

healthcare assistants in Ireland ...............eeeeceececeeeeeeeeeeeeeeeeeeeeeeeeeeesees 38 
8. Modelling skill supply And deMANG ........ ee eee eeeeeeeeeeeeeeeeeeeenaaees 41 
9. Positioning the approaches and tools analysed for closing the 

learning outcomes feedback lOOp ..............::cscccceeeeeeeeeeeeeeeeeeeeeeeeeeesees 50 
10. Objectives of an ERS approach............eececcccececceceetesseceeeeetseeeeeeneees 84 
11. Characterisation of the healthcare assistant level 3 programme 

by the VET provider (Netherlands) ..................c:::eeecceeeeeeeeeeeeeeeeeeees 107 
12. Clusters of skills provided by the VET programme (Netherlands) .. 108 
13. ‘The VET programme has provided the graduates with the skills. 

| believe the graduate acquired this skill’ (Netherlands) ................. 110 
14. Characterisation of all programmes together by the VET 

providers (Lithuania/Netherlands)............:ccceeeeeeseeeeeeeeeeeeeeeteeeeeeeees 113 
15. Clusters of skills provided by the VET programmes 

(both HC and ICT in Lithuania and the Netherlands)............:.06 114 
16. ‘The VET programme has provided the graduates with the skills. | 

believe the graduate acquired this skill’ (Lithuania/Netherlands).... 116 
17. Schematic overview of the feedback loops related to 

developing and renewing qualifications ..............eeeeeeeeeeeeeeeeeeeeeeeneee 142 
Boxes 
1. Final skills reference Point ............::::ccccceeeesseeeeeeeeeeeeeeeeeeeeeeeseneeeeeeeees 13 
2. Key research questions work assignment 3 (WAS) ..........:::cceeeees 28 
3. Prey INNCHINMGNIGS ccsstaresncrncceceanansantonuad anurans se wlaenesennnatrececeinaneusece 32 
4. — Selfie (self-reflection on effective learning by fostering the use of 

innovative educational technologies) .............::ceeceeeeeeeeeeeetteeeeeeeeeeeeeee 86 
5. | ESCO transversal skills and competences and foundations 

clustering, as proposed in November 2019 ..............:cccccccceeeeeeeeeeeeees 88 
6. | Suggestion of a reference point: the Australian list of skills and 

attributes accompanied with reflections ................ccccccccceesceeeeeeeeeeeeees 91 
re Proposed reference POM cccciccssiccenencitsctereateasescinenenteardhsnnmcnceeineneiseee 96 
8. Reference point used for the pre-test of the ERS... ee 99 


Review and renewal of qualifications: 
towards methodologies for analysing and comparing learning outcomes 


Executive summary 


Aim of the study 


The feedback between vocational education and training (VET) and the labour 
market can provide important input for the review and renewal of qualifications. A 
feedback loop that is based on learning outcomes helps provide deeper insights 
into what is required on the labour market, what is offered in training provisions 
and assessed at the end of a learning programme. It can also provide insights into 
how the learning outcomes achieved by qualification holders are perceived in the 
labour market and, in particular, by their employers. This type of feedback can help 
to better shape the specific profile of qualifications and so provide important 
information for qualifications authorities and for providers offering these 
qualifications. The use of learning outcomes is crucial in this approach: it not only 
allows to identify (new) skill needs in the labour market, but also to reflect on the 
learning outcomes acquired with a specific qualification as they are realised in the 
workplace. The feedback loop on learning outcomes suggests continuous dialogue 
on intended and achieved learning outcomes, trying to improve the stated 
expectations (intended learning outcomes) on the basis of actually achieved 
outcomes as applied and perceived in the labour market. In this context, the crucial 
question is what data are available to complete the feedback loop or how such 
data can be collected. 

The aim of this study is to contribute to strengthening the quality and relevance 
of qualifications and completing the feedback loop between education and the 
labour market. It examines methods of collecting data on the match/mismatch 
between qualifications and labour market requirements, including analysis of how 
achieved learning outcomes are applied and perceived in the labour market (for 
example methods of collecting the experience of employers with holders of these 
qualifications). This report addresses the following two questions: 

(a) which data already exist in the countries, providing insight into the relevance 
of qualifications to employees, employers and other labour market 
stakeholders? 

(b) how can survey methodology be designed to systematically capture the 
experiences and appreciations of employers as regards the content and 
profile of qualifications? To what extent, based on limited testing, can 
scalability of the methodology be achieved? 

This summary is structured in line with these questions and completed by 
conclusions and recommendations. 
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Exploration of methods for completing a feedback loop 


Countries have different mechanisms for reflecting on the match between the 
supply of VET programmes and qualifications and labour market demand. A first 
mechanism, often informal, is that VET providers have direct contact with 
employers, for instance related to work placements, internship or apprenticeships. 
Quality assurance mechanisms may also include references to tracking graduates 
and employer satisfaction. This allows VET providers and employers to discuss 
whether the content of the qualifications and the delivery of the programme are in 
line with labour market needs. Second, the supply-demand matching mechanism 
can be integrated into the VET governance structure. Labour market stakeholders 
are (partly) responsible for the VET system and therefore systematically reflect on 
the content of VET programmes. This can be institutionalised, for example in sector 
skills councils or trade committees. Third, specific research approaches can be 
applied to gather information on the relevance of VET qualifications and the 
content of VET programmes. The following distinction can be made for data 
sources and approaches, particularly those offered at European level: 

(a) skill mismatch analyses, especially Cedefop’s European skills and jobs survey 
that asked adult employees how their skills and qualifications match the needs 
of their jobs; 

(b) vacancy analyses, especially Cedefop’s big-data analysis of job vacancies 
that provides insights into the competences desired on the labour market; 

(c) forecasting approaches that are used for reviewing and updating 
qualifications, particularly Cedefop’s skills forecast; 

(d) VET graduate tracking measures that provide quantitative and/or qualitative 
information about the labour market outcomes (destination, employment 
status, occupation and/or satisfaction levels of both VET learners and 
employers) for graduates of certain programmes. These findings can be linked 
to more specific (types of) learning outcomes associated with certain 
qualifications. Information obtained in such surveys is used to renew 
qualifications and programmes and to provide advice to new students on what 
career changes VET programmes offer; 

(e) employer reflection surveys that can clarify the more specific demands for 
learning outcomes of qualifications. These measures and surveys allow 
mapping variation in demands across sectors, regions, or Member States. 


Most approaches explored provide important data for creating skills 
intelligence but are not sufficient for completing the feedback loop based on 
learning outcomes: 


Executive summary 


skill mismatch analyses and the European skills and jobs survey provide 
information on the degree of match between skills supply and demand, but 
usually at a higher aggregated level and not referring to individual 
qualifications; 

online vacancy advertisements, including Cedefop’s skills online vacancy 
analysis tool for Europe (skills-OVATE), can be a valuable source to see what 
kind of skills and competences are in demand in a particular occupation. 
However, this does not indicate whether or not these skills are provided in the 
education programmes that prepare for this occupation. It can also be 
questioned whether it is at all possible to link vacancy data to specific 
qualifications and whether this can provide relevant information on graduates' 
skills presence; 

forecasting procedures at national level and Cedefop's pan-European skills 
forecast provide insights into general future trends and do not typically relate 
to the perception of the learning outcomes achieved and realised by 
graduates. Further, the European skills forecast does not refer to individual 
qualifications but uses broad categories; 

VET graduate tracking surveys are a valuable source for tracking the further 
development of graduates after completing their VET programme, and so can 
provide an indication of whether the qualification is in demand on the labour 
market, and sometimes even which skills and competences have contributed 
to their career development. However, many of these surveys and studies 
focus on general issues, such as whether graduates find a job easily or 
whether they were satisfied with the training received. The extent to which 
these surveys and analyses are able to capture the demand for specific 
learning outcomes and the orientation and profile of qualifications in detail is 
not yet fully explored. Graduate surveys are usually not able to capture a 
perspective on whether the intended learning outcomes are actually achieved. 


One approach that is not systematically implemented in European Member 


States, but might have the potential to provide the reflection on individual learning 
outcomes, is the employer reflection survey that takes into account the content of 
qualifications. These surveys are aimed at employers, to measure their satisfaction 
with VET graduates working in their company in terms of the skills and 
competences they have acquired and the use of these skills in the workplace. Only 
a few of the 10 countries surveyed have employer reflection surveys that look at 
the content of qualifications. In some countries this was due to the absence of any 
surveys addressed to employers, while in cases where employers were addressed, 
the type of survey generally tended to focus on identifying skill needs rather than 
on gaining insights into their satisfaction with learning outcomes. 
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Challenges related to employer reflection survey concerns and the content of 
qualifications are too heterogeneous to be assessed in a generic employer 
reflection survey (ERS) tool: qualifications have weak links with occupations and 
the labour market is not accustomed to reflect on the content of qualifications; the 
outcomes are compromised by high levels of subjectivity and low response rates; 
and ERS can be costly. Five of the national experts stated that ERS are regarded 
as useful in their country because of the opportunities they provide but, in some 
countries, the national expert questioned whether such ERS would at all be useful 
and desirable in their country contexts. It may be that the need to have ERS is 
limited in systems where there are strong governance and system links between 
the labour market and the VET system, where VET qualifications are governed 
and delivered in close cooperation with employers. It is likely that in more countries 
there are already (more direct) feedback loops within the system, to tailor the VET 
provision to the emerging economic and societal needs. The ERS might therefore 
be more relevant for Member States where the VET system is operating within a 
relative arm’s length of the labour market. 

An observation is that existing approaches for identifying — skills 
match/mismatch are strongly biased towards labour market demand and often do 
not consider the supply side, the provision of education and training. VET graduate 
tracking and employer surveys can be exceptions to this. 

Figure 1 presents the positioning of the approaches and tools analysed related 
to their focus: demand or supply side, on the one hand, and individual qualifications 
and learning outcomes or broader categories on the other. 

It can be concluded that a more promising approach for completing the 
feedback loop seems to be the use of surveys which can ask whether employers 
are generally satisfied with graduates or whether they experience shortages, or 
what is the demand for specific qualifications in the labour market. Employer 
surveys exploring their satisfaction with their employees’ learning outcomes are 
the most direct means of tracking the link between intended and actual or realised 
learning outcomes, as they ask employers whether recent graduates can actually 
apply the skills and competences that were promised in their qualification. 
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Figure 1. Positioning the approaches and tools analysed for closing the learning 
outcomes feedback loop 
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Source: Cedefop. 


Survey methodology to capture employer reflections on 
achieved learning outcomes 


The objective of the employer reflection survey approach is to support VET 
providers to match their offers better to the needs of their direct labour market 
stakeholders. It offers VET providers a possibility to engage better with their 
beneficiaries (both graduates and employers) and reflect together with them on the 
outcomes of the VET programme offered, also reflecting on the regional labour 
market they mostly serve. Hence, the ERS prototype is not an accountability tool 
but rather a tool to initiate dialogue between VET providers and employers (and 
graduates). However, the data collections should allow aggregation of the data to 
sectoral, national and cross-national levels. The main opportunities and objectives 
of an ERS approach are schematically presented in Figure 2. 
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Figure 2. Opportunities and objectives of an ERS approach 
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Methodological condition: 
A skills reference point that is specific and meaningful at provider level, 
but allows aggregations to national/sectoral and cross-national/sectoral level 


Source: Cedefop. 


The workflow of the ERS in the pre-test in Lithuania and the Netherlands was 


as follows: 


(a) 


step 1: the VET provider was selected to test the ERS prototype. The 
approach and content of the questionnaires was discussed with 
representatives of the VET provider who provided a characterisation of the 
VET qualification in terms of a learning outcomes profile in relation to the 
reference point for the it (questionnaire for VET provider); 

step 2: the VET provider sent the link to the questionnaire (questionnaire for 
graduates) to the graduates. In the invitation email, the researchers were also 
introduced. The researchers also asked the VET providers to send out 
reminders. Although the VET provider was responsible for inviting the 
graduates, the individual data gathered were not available to the provider; 
step 3: the VET provider compiled a list of employers that potentially hired 
graduates from the programme and sent the invitation link to participate in the 
survey to the employers (questionnaire for employers). This survey also 
included questions on whether the employer consented to being contacted by 
the researchers to discuss the aggregated results; 

step 4: the researchers analysed the survey results and discussed the 
outcomes with the VET providers and employers to start a dialogue on 
intended and realised learning outcomes and the content of the VET 
qualification. 
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The relevant ERS approach has three questionnaires: one for the VET 
providers, one for the graduates and one for the employers. In these 
questionnaires, when focusing on questions related to learning outcomes, it was 
agreed to use a skills typology that is broad enough so that it can be used 
regardless of the sector, occupation or qualification in question but still allows for 
respondents to score several specific items for each category included. 

The final list of learning outcomes used in the pre-test is presented in Box 1. 


Box 1. Final skills reference point 


A. General occupation-related skills and competences as exercised in the 
workplace 


This cluster describes some general occupation-specific skills and competences: 
(a) applying professional knowledge to job tasks; 

(b) using technology effectively; 

(c) applying technical skills in the workplace; 

(d) maintaining professional standards; 

(e) observing ethical standards; 

(f) using research skills to gather evidence. 


B. Specific occupation-related skills and competences as exercised in the 
workplace 


For information and communications technology (ICT) service technician: 
provide technical documentation; 

) perform ICT troubleshooting; 

use repair manuals; 

configure ICT system; 

administer ICT system; 

maintain ICT server/system; 


a  ——— 
Se @ (ef @ fe 
~~ — ~— 


(g) perform backups; 

(h) repair ICT devices; 

(i) implement ICT recovery system; 
(j) manage ICT legacy implication; 


= 
~~ 
co 


use precision tools. 


For healthcare (HC) assistant: 

a) monitor basic patient signs; 

b) communicate with nursing staff; 

c) empathise with the healthcare user; 
d) interact with healthcare users; 

e) provide basic support to patients; 

f) identify abnormalities; 

g) support nurses; 

h) ensure safety of healthcare users; 


( 
( 
( 
( 
( 
( 
( 
( 
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( 
( 


i) convey medical routine information; 


i) 
(j) manage healthcare users’ data; 


k) conduct cleaning tasks. 


C. Teamwork and interpersonal skills as exercised in the workplace 
This cluster describes how the graduate works in an interpersonal context: 


( 
( 


a) working well in a team and working collaboratively with colleagues to complete 
tasks; 

b) getting on well with others in the workplace and understanding different points of 
view; 

c) ability to interact with colleagues from different or multicultural backgrounds; 

d) ability to follow instructions; 

e) ability to instruct and/or lead others; 

f) ability to handle conflicts. 


D. Employability and enterprise skills as exercised in the workplace 


This cluster describes how the graduate works in an organisational context and in the 
labour market: 


( 


( 
( 
( 
( 
( 
( 
( 
( 


( 
( 
( 
( 
( 


( 


a) ability to work under pressure; 

b) reflect work processes and procedures; 

c) capacity to be flexible in the workplace; 

d) ability to meet deadlines; 

e) understanding the nature of your business or organisation; 

f) ability to manage processes/projects; 

g) taking responsibility for personal professional development (keep up to date); 
h) demonstrating initiative in the workplace and show sense of initiative; 
ability to solve problems; 

oral communication skills; 

k) written communication skills; 

1) foreign language skills; 

m) working with numbers; 

n) persistence and endurance; 

0) critically reflect on own role and place in society. 


Source: Cedefop. 


The pre-testing resulted in the following lessons learned: 


(a) the pre-test faced severe challenges in reaching out to employers and 


graduates. While the VET providers approached could be convinced to 

participate rather easily, it was more challenging than expected to have them 

reach out to employers and graduates. There are several reasons for this: 

e reaching out to graduates required VET providers to have a functioning 
alumni policy in which contact details of graduates are kept up to date 
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and also assured that graduates provided consent in being approached 
for surveys after obtaining their VET qualification; 

e the route from graduates to employers (i.e. asking graduates to provide 
contact details of their supervisors) proved to be a dead end in the pre- 
test. In the rerun of the pre-test, the approach was changed to contacting 
employers directly. For this purpose, the research team asked the VET 
provider to make an inventory of which employers could have hired 
graduates in the last two years. Many employers that responded to the 
survey invitation were excluded, as they did not recently hire graduates 
from the specific VET programme; 

the questionnaires and the skills lists used in the pre-test seem to be 

appropriate. The skills lists are detailed enough to allow in-depth reflections 

on the content of the qualifications but are short enough (38 items) to be used 
in a survey without burdening the respondents with tiresome lists of skills. This 
points to a good balance of complexity and simplicity of the reference point 

used. The average time for the employers to complete the full survey was 15 

minutes. The structure with the four clusters worked well and allows a 

maximum of comparisons between different qualification profiles, while 

allowing occupation-specific skills to be assessed as well. The occupation- 
specific list (cluster B) seemed to work better for the healthcare assistant than 
for the ICT technician; the skills listed for the latter were less recognised by 

VET providers, graduates and employers; 

there is too little data to establish a lesson learned about comparing the results 

for different qualification profiles and the results between countries. At the time 

of drafting this report, sufficient data were available only for one VET 
qualification offered by one VET provider. Therefore, no comparison across 

VET providers, qualification profiles and countries could be conducted. Also, 

it was not possible to benchmark one qualification against others. As data 

collection efforts still continue, the research team hopes to obtain additional 
data that allow such analyses; 

despite the lack of data, the ERS is an interesting and promising approach to 

closing the feedback loop, looking in detail at the content of qualifications. It 

allows more direct exchange between VET providers and the employers of 
graduates on whether what is envisaged in terms of learning outcomes is 
perceived as being present in the work environment. 
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Conclusions 


The study reached the following conclusions: 


(a) 


there are valuable sources for the creation of skills intelligence, but existing 
methods and tools rarely relate to individual qualifications and usually do not 
consider actual learning outcomes as they are realised and perceived in the 
labour market. They often exclusively focus on the demand side and do not 
consider the supply of education and training sufficiently, so they are of limited 
use for completing the feedback loop; 

employer reflection surveys have the potential for completing the feedback 
loop based on learning outcomes but they are usually not conducted in a 
systematic way at national level. The examples identified show a variety of 
different approaches and there are several challenges associated with their 
design. Their usefulness in illustrating the match between the intentions of the 
VET system and the demands of the labour market seems to be assessed as 
higher in countries with weaker governance and system links between the 
labour market and the VET system; 

the list of skills included in the reference point, as developed and used for the 
pre-test of the ERS, seems appropriate insofar as it allows for reasonable 
skills assessments, striking a balance between occupation-specific and 
generic skills, while Keeping the total number of skills at a manageable level 
to allow their application in employer and graduate surveys. In addition, the 
balancing act of developing a reference point — based on existing skillsets — 
while avoiding both oversimplification (with the risk of being potentially 
meaningless) and over-complexity (with the risk of not being understood by 
graduates and employers) seems to have been achieved; 

the pre-tested ERS approach is promising but also challenging, as it requires 
VET providers to keep track of their graduates (alumni-policy) and of the 
employers of their graduates. This is a challenge, especially in sectors with 
many small and medium-sized enterprises (SMEs); 

the tested ERS approach is potentially scalable in terms of using the 
questionnaire for other qualification profiles, engaging other VET providers 
and offering different language versions. 
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Recommendations 


The study findings and conclusions have led to the following recommendations: 


(a) 


(b) 


it is recommended to integrate methods to complete the feedback loop more 
strongly into VET governance and quality assurance procedures and 
structures; 

it is recommended to combine different sources for completing the feedback 
loop, as all sources have their specific added value and advantage; 

it is recommended not just to rely on national- or system-level feedback loops 
to gather information for the renewal of VET qualifications, but to strengthen 
more direct feedback loops between VET providers and their (local or 
regional) labour market stakeholders so that reflections from the labour 
market can be taken on board in quality assurance and curriculum renewal 
more directly; 

it is recommended to experiment further with the employer reflection survey 
methodology and try to set up an infrastructure inspired by self-reflection on 
effective learning by fostering the use of education technologies such as the 
‘Selfie-tool’ to capture the opinions and experiences of school leaders, 
teachers, students and (in work-based learning settings) companies. 


The following aspects need to be considered in the continued testing: 

e expand to new countries. The ERS has been pre-tested in the 
Netherlands and Lithuania, but it would be beneficial to obtain insights 
into whether the approach could also work in other VET contexts; 

e expand to more VET providers willing to participate and assure buy-in at 
national level and within economic sectors to test the approach; 

e expand to other qualification profiles besides the healthcare assistant and 
the ICT technician. The pre-test looked at two carefully selected 
occupations but is very limited in covering the full scope of VET 
qualifications. More insights are needed into how the ERS approach 
works in different occupational contexts; 

e combine the testing of the ERS with support to VET providers to develop 
policies on employer engagement and alumni policies; 

e combine the ERS outcomes at national or system level with information 
from other sources closing the feedback loop (vacancy analysis, skills 
forecasts) and discuss the combined findings with VET providers and 
employers. 


17 


CHAPTER 1. 
Background of the study, key objectives and 
methodological approach 


1.1. Setting the scene: relevance of qualifications and 
feedback loops 


Strengthening the quality and relevance of vocational education and training (VET) 
qualifications is essential for ensuring employability of individuals and 
competitiveness of labour markets, particularly when considering developments, 
such as the rapidly changing technological environment of VET and the associated 
rapidly changing skill needs. This topic is at the core of European policies. For 
example, the European skills agenda (European Commission, 2020a) emphasises 
the improving skills intelligence to deliver training relevant for the labour market, 
and the Council recommendation on vocational education and training for 
sustainable competitiveness, social fairness and resilience (Council of the 
European Union, 2020) calls for vocational education and training that is agile in 
adapting to labour market changes, as well as for a regular update of qualifications. 
In order to enhance the quality and relevance of VET qualifications, itis necessary, 
on the one hand, to have a good understanding of their content and, on the other 
hand, to employ reliable methods and approaches for identifying the skills needed 
on the labour market. 

Methodologies for analysing the learning outcomes included in VET 
qualifications and mapping them to a reference point, as developed and tested in 
the previous parts of the Comparing VET qualifications study, can help to 
understand better the content of qualifications. The comparison of VET 
qualifications and their learning outcomes within or across different Member States 
can yield relevant insights into the differences and similarities of the contents of 
qualifications and inspire their renewal. However, qualifications with similar content 
may have a different standing and relevance across different labour markets. This 
may be influenced by broader trends, such as technological changes that reshape 
labour markets, but also more specific trends within Member States, such as the 
historical reputations of certain qualifications. Therefore, approaches focusing on 
analysing the content of qualifications (in terms of learning outcomes) need to be 
complemented by an analysis of how the intentions of the education and training 
system (intended learning outcomes as included in qualifications descriptions) 
match the expectations and needs of employees and employers in a given context, 
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and to what extent they are satisfied with what they get (achieved learning 
outcomes as applied in the work context). This information is an important aspect 
for assessing and improving the relevance of qualifications for the labour market 
and the match with VET provision. 

Processes that keep VET up to date with the labour market — the interaction 
between the worlds of work and of education — have more recently been called 
feedback mechanisms or feedback loops (?). Before presenting the key objectives 
of this study, the main research questions and the methodological approach, 
Sections 1.1.1 and 1.1.2 elaborate on the concept of the feedback loop between 
VET and the labour market and the relevance of qualifications, introducing the 
question of what data are needed to complete the feedback loop and how such 
data can be gathered. 


1.1.1. Feedback loop between VET and the labour market and VET 
governance models 

The Cedefop study Renewing VET provision (Cedefop, 2013) explored the 
functioning of formal feedback mechanisms established to ensure dialogue 
between education and training and the labour market. Such formal feedback 
mechanisms are typically based on a strong (legal, institutional or political) 
foundation, ‘are established on a permanent basis, and comprise two or more 
actors (for example the state, employer and employee organisations)’ (Cedefop, 
2013, p. 7). They can be defined ‘as purposefully implemented formal institutional 
procedures, determining the particular roles of various stakeholders in planned 
renewal of VET provision’ (Markowitsch and Hefler, 2018, p. 287). Compared to 
informal feedback processes, these formal mechanisms are considered as inbuilt 
governance structures. Examples of informal feedback processes include ‘local 
school boards, alumni networks, career fairs, cooperative projects between 
schools and companies, and internships. These informal processes do not 
necessarily result in any formal changes at national level (for example new 
standards or curricula) although they can be equally or even more important to 
keep VET relevant for the society and the labour market than formal ones’ 
(Markowitsch and Hefler, 2018, p. 289). 

Figure 3 presents the basic model of feedback mechanism between VET and 
labour market established at system level. 


(*) See Cedefop, 2013; 2009; Markowitsch and Hefler, 2018. 
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Figure 3. | Basic model of feedback mechanism between VET and labour market 
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Source: Cedefop, 2013, p. 8. 


The Cedefop (2013) study particularly explored ways in which stakeholders 
are involved in these procedures in various areas and at different levels and can 
influence the review of VET provision (qualifications, curricula, teaching and 
learning processes). Based on the analysis of relevant procedures in 15 countries 
(), the study identified four types of feedback mechanism at system level: ‘The 
four types are (Cedefop, 2013, p. 9): 

(a) liberal, characterised by a low degree of coordination, where feedback 
between VET providers and the labour market is mainly regulated through the 
market; 

(bo) statist, characterised by strong state regulation of education and weak links 
between education and the labour market in terms of formal communication; 

(c) participatory, which allows for the participation of social partners in the 
processes, but mainly in a consultative role; 

(d) coordinated, where social partners are the drivers of renewal processes and 
play an active role in its implementation’. 


(3) The study covered the following countries: Austria, Bulgaria, Denmark, Estonia, 
France, Finland, Germany, Hungary, Ireland, Spain, the Netherlands, Poland, 
Slovenia, Sweden and the UK-England. 
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These four types are presented in Figure 4, together with some country 
examples. 


Figure 4. | Four types of formal feedback mechanism between VET and labour 
market 
Liberal model Statist model 
Government/ Government/ 
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NB: White arrows indicate a modest influence, while grey arrows show main interaction. 
Source: Markowitsch and Hefler, 2018, p. 301 (based on Cedefop, 2013). 


The way of involving the labour market stakeholders differs between the four 
models. This is illustrated by the following examples from the countries covered by 
the current study (4). 

The coordinated feedback model can be observed in Denmark and in the dual 
VET in Austria. ‘The coordinated feedback mechanism is characterised by the 
decisive and far-reaching role of organised business and organised labour in the 
renewal processes of VET. Initiative to renew VET content typically comes from 
business interest organisations, with large employers as visible spokesmen in 
favour of a reform or from trade unions that are able to highlight specific issues 
and problems relating to the labour market. These demands are mediated by the 
social partners and expressed to the government in the form of proposals. The 
government then takes the formal decision on any change. The implementation of 
change again depends largely on the social partners who are responsible for major 
parts of the provision (for example apprenticeship places, training of trainers, 
assessment, etc.)’ (Markowitsch and Hefler, 2018, p. 297). In Denmark, systemic 


(*) Austria, Bulgaria, Denmark, Finland, France, Ireland, Lithuania, the Netherlands, 
Spain, UK-England. 


21 


Review and renewal of qualifications: 
towards methodologies for analysing and comparing learning outcomes 


feedback loops between labour market and VET are in place through the trade 
committees, which are directly responsible for drawing up learning outcomes for 
individual qualifications. In Austria, since the social partners play a decisive role in 
the development and governance of the Austrian (dual) apprenticeship system (as 
defined in the governance of the so-called Vocational Training Advisory Board), 
employers’ knowledge of the skills of graduates is also already directly integrated 
into the governance structure. 

The participatory model can be observed in Spain and Finland. In this model, 
the state is responsible for curricula and examinations, while the social partners 
have a formal, consultative role (Markowitsch and Hefler, 2018, p. 296). Thus, the 
social partners are involved in ensuring the relevance of qualifications and their 
learning outcomes. Finland has systematic and regular processes for anticipating 
skill needs, developing skills and evaluating the provision of education. These 
processes collect a wide range of information on vocational qualifications, their 
relevance and the relevance of their learning outcomes. In general, the 
participation of all relevant stakeholders is ensured by legislation. The so-called 
tripartite principle (kolmikantaperiaate) is the basis for the planning, 
implementation and evaluation of VET in Finland: in the design, planning and 
decision-making on VET qualifications, representatives of education, the world of 
work and other relevant stakeholders are involved in the various phases of the 
process. This principle also applies at all levels: national, regional and provider. In 
Spain, the update of professional training degrees is carried out at the national 
level and in some autonomous communities, and is based on qualitative methods 
of work groups, composed of sectoral experts who examine the results of learning 
(of a certain qualification) and propose changes in their formulation. This method 
depends more on the demand or pressure from sectoral lobbies than on analysis 
or quantitative reports on satisfaction or training needs (°). 

The statist feedback mechanism, that can be identified in the Austrian school- 
based VET and in Bulgaria, is characterised by State funds and control. The 
content is usually determined by temporary working groups or commissions set up 
by the Ministry of Education. Social partners are involved but only have an informal 
role. In Bulgaria, the method that is currently used to inform VET programmes is 
through the expert commissions at the National Agency for VET (NAVET), which 
are in charge of writing and reviewing VET standards. The commissions are 


(5) Currently, the sectoral commissions in continuing VET (CVET) are being renewed at 
national level and there are some initiatives to improve the renewal mechanisms of the 
catalogue of training offer in CVET and also in some autonomous communities, but, 
so far in CVET, there is no systematic method of analysis of satisfaction of 
competences by companies. 
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divided into professional areas (°) and are tripartite bodies, including 
representatives of State organisations, employers’ organisations and of workers’ 
organisations. VET standards are generally developed/reviewed by working 
groups of three to five people (mostly including a teacher, a worker and a company 
director). Of note is that sectoral organisations, national employer and employee 
organisations can send proposals to NAVET for the development/review of VET 
standards. So far, a review has been initiated on the initiative of stakeholders, due 
to changes in: 

(a) technology; 

(b) the structure of the labour market; 

(c) skills requirements of employers in a respective professional field, etc. (’). 


In Austrian full-time upper secondary VET, the Ministry of Education is 
responsible for renewing framework curricula; the collection of feedback from 
employers and employees takes place only informally. For example, each school 
is obliged to have a so-called advisory board in which employers should be 
involved (8). Proposals for changed curricula are developed by temporary 
commissions (Lehrplankommission), consisting of VET teachers representing all 
relevant regional VET schools. Typically, a series of forums and conferences 
support the exchange process with social partners, sector experts, and individual 
firms. Participation of any actors within the Lehrplankommission beyond the 
Fachabteilung (°) and VET teachers is voluntary and any hints given remain 
informal. However, even in the absence of a formal role, the strong influence of 
Austrian social partners on curricula reform must not be underestimated. Beyond 
informal participation in the preparation of regulation on curricula, recognised 
social partners typically enjoy the right to consultation on any formal regulation 
prior to implementation (Markowitsch and Hefler, 2018, p. 295). 


(°) A list of the expert commissions is available in Bulgarian: 
http://www.navet.government.bg/bg/nachalo/struktura-na-napoo/ekspertni-komisii/ 


(7) Inits launch of apprenticeships, Bulgaria is working on a broader involvement of labour 
market stakeholders (the methodological framework is not published, but was made 
available through the project manager of the Domino project. See: 
http://dominoproject.bg/en/). 


(8) Although VET schools and colleges have to follow the centralised framework curricula, 
the Ministry of Education has the legal obligation to authorise individual schools to 
issue curriculum regulations within a specified framework in accordance with local 
requirements. However, the autonomy of the individual locations is limited and 
deviations from the prescribed framework curriculum must be justifiable with regard to 
the educational mandate of the individual schools, the transfer possibilities to other 
schools and entitlements. 


(°) Specialised administrative units (Fachabteilungen) within the Ministry of Education are 
responsible for particular occupational fields and its VET colleges and schools. 
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The liberal type feedback mechanism can be found in the UK-England VET 
system, being characterised as ‘an ideal type education market of (individual) 
sellers and buyers’ (Markowitsch and Hefler, 2018, p. 290). In England, further 
education providers align their programmes to established professional norms, and 
the perceived needs of students and employers. In this system, characterised by 
an absence of State regulation, the sector skills councils (SSCs), introduced in 
2003, are the latest attempt to add a ‘formal feedback mechanism’ to the market- 
focused approach (Markowitsch and Hefler, 2018, p. 292). 

In general, such feedback loops (as described in the four types) can lead to 
changes in the content of qualifications, since needs identified on the labour market 
can be integrated into qualifications in renewal processes. In principle, it is also 
possible to use these feedback mechanisms to map the extent to which skills 
required on the labour market are or are not included in the qualifications. 
However, the model presented in Section 1.1.1 is a basic one that can also be 
contested as too simplistic. For example, in addition to such feedback mechanisms 
at system level, there are various other approaches at different levels, such as 
regional, sectoral and provider or qualification. This basic model focuses on the 
intentions of qualifications and does not explicitly address the perception of these 
qualifications on the labour market. 


1.1.2. Learning outcomes, continuous dialogue and necessary data to 
complete the feedback loop 

A feedback loop that is based on learning outcomes helps to get deeper insights 
into what is required on the labour market, what is offered in training provisions 
and assessed at the end of a learning programme. It can also provide insights into 
how the learning outcomes achieved by qualification holders are perceived in the 
labour market and, specifically, by their employers. This type of feedback can help 
to shape the specific profile of qualifications better and therefore provide important 
information for qualifications authorities and for providers offering these 
qualifications. The use of learning outcomes is crucial in this approach as it not 
only allows to identify (new) skill needs in the labour market, but also to reflect on 
the learning outcomes acquired with a specific qualification as they are realised in 
the workplace. Since the latter aspect is not addressed explicitly in the basic model 
presented in Section 1.1.1, this approach can be understood as completing the 
feedback loop. Figure 5 illustrates the feedback loop based on learning outcomes 
which can serve as a key reference point in the dialogue between the worlds of 
work and of education. 
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Figure 5. Learning outcomes feedback loop 
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Source: Auzinger et al., 2017, p. 34 (based on technical specifications). 


The feedback loop on learning outcomes refers to continuous dialogue on 
intended and achieved learning outcomes, trying to improve the stated 
expectations (intended learning outcomes) on the basis of actually achieved 
outcomes as applied and perceived in the labour market. The intended learning 
outcomes are written (possibly jointly) at stage 1 and used to inform the teaching, 
learning and assessment process carried out during the programme (stage 2). 
Assessment can be conducted in various ways (as part of the recognition of prior 
learning, as formative assessment supporting the learning process or as 
summative assessment resulting in the award of a unit or part of a qualification), 
with diverse methods and at different locations involving different types of 
stakeholder. A central aspect is how the assessment requirements are 
communicated to the learner, how learning outcomes are used in this dialogue, 
and how assessment outcomes are used to reflect on the learning outcomes 
included in the qualifications. In stage 3, the learner's achievements at the end of 
the learning process are recorded and assessed on the basis of the learning 
outcomes described in stage 1 and ideally lead to certification and the award of a 
qualification. Here, an important element is how the outcomes of the assessment 
and the achieved learning outcomes are made visible and how this is signalled to 
the labour market and communicated to future employers (for example, are 
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descriptions of the learning outcomes of the qualifications available to the labour 
market stakeholders?). In stage 4, the graduates put their achieved learning 
outcomes into practice. Ideally, the relevance and quality of achieved knowledge, 
skills and competence is monitored and stage 4 provides renewed information for 
writing of learning outcomes, when the experience of employers and also of the 
holder of the qualification is taken into account when qualifications are reviewed 
and renewed. 

A previous study for Cedefop (Auzinger et al., 2017) explored the feedback 
loops related to developing and renewing qualifications in selected countries (the 
same 10 countries as covered by the current project). This study particularly looked 
into the role of learning outcomes and the dialogue with the labour market in the 
development, review and renewal of qualifications. It concluded that labour market 
stakeholders generally play an important role in the reviewing/renewal of learning 
outcomes. This active involvement in most countries goes beyond providing 
feedback and involves directly formulating learning outcomes (for example in 
working groups, expert groups or committees). In many initial VET (IVET) context 
studies, a strong emphasis was also placed on continuous dialogue between the 
worlds of education and work during the whole lifecycle of a qualification and via 
various channels (informal as well as formal ones) ("°). 

The previous study also revealed that in none of the IVET contexts analysed 
was feedback on the achieved learning outcomes of graduates and how they are 
experienced by graduates and employers systematically collected. The study 
found that procedures and mechanisms involved in undertaking reviews were 
more frequently based on direct stakeholder involvement and _ (informal) 
discussions and using learning outcomes, rather than gathering systematic data 
on experiences of graduates and employers. The feedback loops were, therefore, 
considered as only partly completed in the IVET contexts analysed, because the 
fourth stage (taking into account actual or achieved learning outcomes as 
experienced and monitored by employers) seemed to be underdeveloped in most. 

The crucial question is what data are available to complete the feedback loop 
or how such data can be collected. There is a wide variety of tools and methods to 
capture the appreciation of qualifications in the labour market; these approaches 
are part of the processes for creating skills intelligence (''). There are, for example, 


(*°) In Annex 1, Figure 17 provides a schematic overview of the feedback loops related to 
developing and renewing qualifications during the whole life cycle of a qualification. 


("') ‘Skills intelligence is the outcome of an expert-driven process of identifying, analysing, 
synthesising and presenting quantitative and/or qualitative skills and labour market 
information. These may be drawn from multiple sources and adjusted to the needs of 
different users’ (Cedefop, 2019a, p. 1). 
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national school leaver surveys and graduate tracer studies, as well as employer 
surveys and job-vacancy analyses. The surveys and analyses can be conducted 
at different levels (national, regional, sectoral, or provider) and by different 
stakeholders. Further, such surveys and analyses can be conducted for different 
purposes, such as gathering information to renew qualifications, to improve the 
quality of VET delivery, to understand skill needs on the labour market. 
Organisations responsible can be those that draft learning outcomes of 
qualifications, sectoral bodies and employers, public employment services, and 
VET providers. An important aspect, however, is to explore to what extent these 
approaches also focus on individual qualifications and the learning outcomes 
included. 


1.2. Key objectives and main research questions 


The aim of this study is to contribute to strengthening the quality and relevance of 
qualifications and completing the feedback loop between education and the labour 
market. It examines methods of collecting data on the match/mismatch between 
qualifications and labour market requirements, including analysis of how achieved 
learning outcomes are applied and perceived in the labour market (through, for 
example, methods of collecting the experience of employers with holders of these 
qualifications). 

This report addresses the questions and issues presented in Box 2. 

Given the feedback loops and mechanisms in place to inform renewal 
processes for the content (learning outcomes) of qualifications, the crucial question 
is what data are available to complete the feedback loop or how can such data be 
collected. Taking this question as starting point, the research questions 1.1-1.3 
follow logically in the sense that they aim at finding out what approaches, sources 
and data are used to understand better the relevance of the learning outcomes 
included in the qualifications and achieved by holders of VET qualifications. As is 
explained in Chapter 2, the study looked at skills mismatch analyses, vacancy 
analyses, forecasting approaches, VET graduate surveys, and employer reflection 
surveys. The research question 2 (2.1-2.4) continues to explore the employer 
reflection surveys for two reasons. First, this is a less-travelled road to capture 
appreciation of achieved learning outcomes of qualifications; second, it might 
provide a more direct closure of the learning outcomes feedback loop compared 
to the other — more indirect — approaches. Research question 3, as a follow-up, 
concerns testing a survey approach with employers and graduates and learning 
from this experience in terms of scalability. 
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Box 2. Key research questions work assignment 3 (WA3) 


1. Which data already exist in the countries providing insight into the relevance of 
qualifications to employees, employers and other labour market stakeholders? 


1.1 To what extent can (existing) VET-graduate surveys illustrate the match between 
the intentions of the VET system and the demands of the labour market? 


1.2 To what extent can (existing) employer surveys illustrate the match between the 
intentions of the VET system and the demands of the labour market? 


1.3 Which data sources, beyond VET-graduate and employer surveys, can illustrate 
the match between the intentions of the VET system and the demands of the 
labour market? 


2. How can survey methodology be designed to capture systematically the 
experiences and appreciations of employers of the content and profile of 
qualifications? 

2.1. Which could be the key target groups for a survey? 

2.2. How could the methodology balance depth of questions and numerical 
coverage? 

2.3 How can survey methodology support international comparison? 

2.4 How can survey methodology achieve understanding of short-term labour market 
outcomes of qualifications (getting a job that matches candidate skills) and long-term 
outcomes (career progress, wage gains, ability to develop skills further)? 


3. To what extent, based on limited testing, can scalability of the methodology be 
achieved? 


Source: Tender specifications. 


1.3. Methodological approach and overview 


1.3.1. Methodological approach 

The focus of the report is on exploring which methodologies and data exist for 
assessing whether the content and profile of qualifications, as provided by the VET 
sector, are appreciated on the labour market, and whether graduates have 
achieved learning outcomes the labour market asks for. Hence, this research study 
consists of the following building blocks and methodological steps (Figure 6), which 
is discussed in more detail in the remainder of Chapter 1. 
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Figure 6. Analytical framework and methodological steps 


Completing the learning outcomes feedback loop: 
gathering insights in the relevance of the content (learning outcomes) of qualifications 
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and small- 
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Source: Cedefop. 


Following this analytical framework, research activities were conducted at EU 
level (desk research, interviews with key experts/stakeholders), as well as at 
country level, in 10 countries (Austria, Bulgaria, Denmark, Finland, France, Ireland, 
Lithuania, Netherlands, Spain, UK-England). Based on the information gathered, 
the research team developed a prototype for testing an employer reflection survey 
(ERS) in two countries (Lithuania, Netherlands) for two selected qualification 
profiles. It was agreed to use the same two profiles for testing as in the first part of 
the overall study: 

(a) healthcare (HC) assistant; 
(6) information and communications technology (ICT) service technician. 


The research activities and methodological steps are further discussed in 
Sections 1.3.1.1 to 1.3.1.4. 


1.3.1.1. Desk research on existing tools, methods and data sources 
Desk research was carried out centrally and at country level. 

The central desk research focused on what is already known about sources, 
existing tools and approaches (including employer surveys) for analysing the 
relevance of skills and competences in the labour market. Within a number of 
projects and studies (such as managed by Cedefop or the European Commission), 
a lot of work has already been done on mapping existing practices for gathering 
skills intelligence: examples include VET graduate tracking measures, skills 
mismatch analyses, real-time labour market information and employer reflection 
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surveys. The outcomes of these projects and studies were taken into account to 
discuss what is already known, what data are already available and to what extent 
such data can contribute to completing the feedback loop. 

At country level, the desk research focused on identifying specific types of 
employer reflection and satisfaction surveys, as well as on understanding why they 
might not be available in a country and discussing other sources used for ensuring 
the relevance of qualifications and their learning outcomes. Within the scope of this 
study, the term employer reflection surveys refers to surveys that aim to provide 
insight into employers’ reflections on the skills and competences of VET graduates 
(after having obtained the qualification). These surveys are aimed at employers, 
seeking their reflections on the VET graduates working in their company in terms 
of the skills and competences they have acquired and the use of these skills in the 
workplace. Such surveys ask about the intended and/or achieved learning 
outcomes of qualifications, or which skills/competences are most (and least) 
relevant to the jobs of graduates, as well as the extent to which the qualification 
has adequately prepared graduates for their current jobs. This excludes the more 
general surveys, such as those aimed at simply giving feedback on how well 
graduates from a particular institution or programme are received (in terms of how 
good they are at doing the job) — without specifying the required and desired skill 
sets, or aspects, related to identifying skills mismatches. 

The term employer satisfaction survey was originally used but this tends to 
see the employer as the one at the receiving end of what the VET system offers, 
rather than as a partner for the joint review and renewal of VET qualifications. 
Therefore, the term employer reflection survey was chosen to emphasise the 
active role of employers in this process. 

The main focus was on surveys of employers with regard to graduates of 
IVET; however, where there were employer reflection surveys focusing on 
continuing VET (CVET) graduates, or graduates of higher VET programmes (for 
example offered by universities of applied sciences), these were also considered. 
In each country, existing practices were discussed in connection with the two 
selected profiles (if available). The information collected was recorded in 
templates; a general one, focusing on the national level, as well as one for each 
ERS selected for the inventory, indicating key aspects of the survey (such as levels 
of implementation, method, coverage). Table 1 presents the main structure of the 
template for collecting information in the 10 countries. 
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Table 1. Template for desk research at country level 


Part 1 
Identification of 
employer reflection 
surveys 


Part 2 

Examples of 
employment 
reflection surveys 


Part 3 
Reflection on 
design features 


Source: Cedefop. 


Please provide a description of how the information used for part 2 
(i.e. examples that fit the criteria as explained above) was obtained. 


If no fitting examples were identified, which other examples were identified 
that did not meet the criteria? 


A. Name of survey 


B. Design and methods 
(including questions concerning organisations responsible, main objective, 
targeted population, general approach, etc.) 


C. Content of the questionnaire 
(including questions concerning unit of analysis, skills classifications used, 
contextual factors included in the questionnaire) 


Including questions concerning the usefulness of employer reflection 
surveys, most suitable methodological approach, preferred level of detail 
of data, own reflections into which design features would be most suitable 
for systematic mapping of the relevance of qualifications 


1.3.1.2. Interviews with key informants 


The overview gained via desk research was complemented by interviews with 
thematic experts and national VET representatives in the 10 countries. The 
interviews had a dual purpose: 

(a) to complement the inventory with practices not identified in the literature 


review; 


(ob) to discuss design features for an ERS. 


The interviews with national stakeholders also explored the feasibility of 
conducting the testing of a to-be-developed prototype. The experts presented in 
Box 3 played a role as key informants. 
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Box 3. Key informants 


EU level: 

(a) experts involved in Cedefop projects on skills mismatch, skills need forecast and 
big-data analysis from online vacancies, European company survey, skills 
surveys; 

(b) experts in job vacancy analyses. 


National level (in 10 countries): 

(a) national VET associations (where relevant); 

(b) national VET government representatives or authorities/organisations involved in 
skills analysis in the labour market; 


(c) persons responsible for an employer reflection survey (if needed to gather further 
information on the survey). 


A total of 32 persons were consulted at national level. 


Source: Cedefop. 


1.3.1.3. Analysis and design of the survey methodology 

Based on the desk research and the key informant interviews, a draft prototype of 
a survey methodology was developed, along with the workflow for its 
implementation. A pre-test of the developed prototype was carried out in Lithuania 
and the Netherlands because these countries already had some experience with 
ERS. Table 2 presents the qualifications that were considered for the testing of the 
prototype. 

The ERS was based on an online questionnaire with translations into the 
national languages. National researchers supported the pre-test and liaised with 
the national stakeholders and institutions that were involved in this exercise. The 
steps taken, any challenges and reflections on the approach based on feedback 
from stakeholders involved were documented by the country researchers. 


1.3.1.4. Finalisation of the prototype and further reflections 

Following the pre-test phase, the core research team analysed and compared the 
data gathered, identified lessons learned and ways to improve the prototype. Final 
reflections were dedicated to issues of scalability of the method applied, the added 
value of comparing results across qualification profiles and countries and on 
potential ways forward. 
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Table 2. Countries and qualification profiles 


Healthcare assistant | EQF level | ICT service technician EQF level 


Service engineer of information 
LT Carer/social worker 3 and communication 4 
technologies 


Healthcare provider 

NL (nursing and 3 
convalescent homes 
and home care) 


ICT management assistant 3 


NB: EQF stands for European qualifications framework. 
Source: Cedefop. 


9123:2: Overview 

The report comprises the following chapters: 

(a) Chapter 2 elaborates on methods for completing the feedback loop and 
discusses data sources providing insight into labour market needs and their 
potential role in assessing the relevance of qualifications. Skills mismatch and 
job-vacancy analysis, as well as forecasting approaches, are considered, and 
particular focus is put on graduate tracking measures; 

(6) Chapter 3 focuses on employer reflection surveys and explores their 
existence, as well as reasons for non-existence in the countries covered by 
this study. Selected surveys are analysed in detail to inform the design of the 
prototype to be tested; 

(c) Chapter 4 is dedicated to the development and the pre-test of the prototype 
of an employer reflection survey. It presents its design features and the 
rationale for the choices made, the workflow for the pre-test in two countries 
(Lithuania and the Netherlands), the results and the lessons learned; 

(d) Chapter 5 draws conclusions related to the research questions and presents 
policy recommendations. In particular, it discusses the added value of bringing 
results of different approaches (such as vacancy analyses and employer 
reflection surveys) together. 
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2.1. Introduction 


Chapter 2 discusses data sources providing insight into the relevance of 

qualifications (and their learning outcomes) to employees, employers and other 

labour market stakeholders, illustrating the match between the intentions of the 

VET system and the demands of the labour market. Emphasis is placed on 

methods and approaches to complete the feedback loop: to close the gap by 

focusing on the learning outcomes achieved as experienced in the labour market. 
The following distinction can be made for data sources and approaches — 
particularly those offered at European level — which are discussed in more detail 

in the remaining part of Chapter 2: 

(a) skill mismatch analyses, particularly Cedefop’s European skills and jobs 
survey, that asked adult employees how their skills and qualifications match 
the needs of their jobs; 

(b) vacancy analyses, particularly Cedefop’s big-data analysis of job vacancies, 
that provide insights into the competences desired at the labour market; 

(c) forecasting approaches that are used for reviewing and updating 
qualifications, and especially Cedefop’s skills forecast; 

(d) VET graduate tracking measures that provide quantitative and/or qualitative 
information about the labour market outcomes (destination, employment 
status, occupation and/or satisfaction levels of both VET learners and 
employers) for graduates of certain programmes. These findings can be linked 
to more specific (types of) learning outcomes associated with certain 
qualifications. Information obtained in such surveys is used to renew 
qualifications and programmes and to provide advice to new students on what 
career changes VET programmes offer ('°); 

(e) employer reflection surveys that can clarify the more specific demands for 
learning outcomes of qualifications. 


(12) See, for instance, the studiebijsluiter in the Netherlands, providing for all VET 
programmes in a systematic overview information about career changes, the usual 
study duration, follow-up to higher education, etc.: 
https://www.s-bb.nl/onderwijs/studie-cijfers 
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2.2. Skill mismatch analyses and the European skills 
and jobs survey 


Skill mismatch refers to situations where skill supply and skill demand diverge, and 
to various types of skill gaps or imbalances resulting; examples include 
unemployment, recruitment difficulties, skills becoming outdated and people doing 
jobs not using their potential. European countries have implemented various 
policies and tools that aim at mitigating skills mismatch and ‘the social and 
economic costs linked to the waste of skills and human potential it entails’ 
(Cedefop, 2018, p. 12). Cedefop offers an information tool, the matching skills 
database, that showcases a collection of innovative policy instruments from EU 
Member States that use information on labour market trends and anticipated skill 
needs to inform and shape upskilling or other skills matching policies for the current 
and future world of work ('°). 

The European skills and jobs (ESJ) survey is the first survey on skill mismatch 
carried out in the EU-28 Member States and it is the first to look at skill mismatch 
over time. The survey asked 49 000 adult employees (aged 24 to 65) across all 28 
Member States how their skills and qualifications match the needs of their jobs. A 
mixed methodology was applied for collecting data: ‘The data were collected using 
quota sampling, following extensive testing, including a pilot survey in January 
2014. The fieldwork (mixed mode, online plus telephone interviewing) was 
conducted from March to June 2014’ (Cedefop, 2015, p. 14). 

The ESJ survey examined drivers of skill development and the dynamic 
evolution of skill mismatch in relation to the changing complexity of the tasks and 
skills required in people’s jobs. It examined to what extent individuals’ qualifications 
and skills are matched to the changing skill demands and complexities of their jobs 
and to what extent employees’ skills are developed and used in their workplace. 
The following aspects are considered as the novel elements and the value-added 
of the ESJ survey (Cedefop, 2015, p. 14): 

(a) ‘the ESJ survey provides the first comparable evidence on skill mismatch 
across all EU-28 Member States from a longitudinal perspective; 
(b) it provides measures of qualification and skill mismatch in total for each EU 

Member State, but also for 11 specific cognitive and non-cognitive skills; 

(c) it collects information on VET and work-based learning (WBL) and examines 
their effect on skill mismatch; 

(d) it draws critical contextual information on the motives, constraints and 
preferences affecting the job choice of individuals; 


(8) https://www.cedefop.europa.eu/en/tools/matching-skills/ 
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(e) it examines the impact of job mobility (occupational, geographic) on skill 


mismatch; 


(f) it looks at the determinants of individuals’ skill accumulation in jobs 
(formal/non-formal/informal training); 

(g) it allows comprehensive analysis of the evolution of skill mismatch by 
measuring the impact of dynamic changes in skills and job tasks within 


workplaces.’ 


The ESJ survey questionnaire was split into eight sections as presented in 


Table 3. 


Table 3. Overview of questionnaire structure 


Selection criteria 


Current job 


Education attainment 


Skills and job demands 


Situation since start of 
current job 


Training and learning 


Situation before current job 


Background information 


Source: Cedefop, 2015, p. 108. 


Content of section 


Questions on age, gender and employment status to confirm that the 
respondent is eligible to take part 


Questions including occupation, industry, type of organisation, changes in 
role since starting job, contract type 


Questions on the respondents’ highest level, type and subject of 
education and assessment of whether it was needed to get their current 
job 


Questions on the type, level and importance of skills respondents require 
to do their current job 


Questions about respondents’ development of skills and the demands of 
their job since they started their job 


Questions on type and reasons for training, and attitudes to learning 


Questions on employment status before starting current job, reasons for 
accepting current job, and level of skills in current job 


Demographics 


While the ESJ survey provides valuable ‘insight into the dynamics of 
qualification and skill mismatch in the EU, focusing on the interplay between 
changes in the (cognitive and noncognitive) skills of employees in their jobs, as 
well as the changing skill needs and complexities of their jobs’ (Cedefop, 2015, p. 
15), it does not refer to specific qualifications. Further, it exclusively focuses on 
skills demand in the labour market and does not consider the supply of skills in 
education and training, making the ESJ survey of limited use for closing the 
learning outcomes feedback loop. 
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2.3. Vacancy analyses and Cedefop’s skills online 
vacancy analysis tool for Europe 


The use of online job vacancy (OJV) portals has significantly increased over recent 

decades, so countries often use OJV analyses to collect information about current 

skill needs and other job requirements (Cedefop, 2019b, p. 9). For example, web- 
spidering programmes can systematically map these vacancies and list the desired 
competences ("). 

OJV analyses can generally add more clarity to questions, such as (Cedefop, 

2019b, p. 26): 

(a) ‘for which occupations is demand increasing most? In which sectors or 
regions? 

(b) what profiles are employers seeking to recruit in these top jobs? (15) What new 
types of job are emerging? For which type of jobs and skills are employers 
facing recruitment difficulties? 

(c) what are employers’ demands for specific skills in specific jobs? How does 
this differ across countries, regions or sectors? What new skills are employers 
demanding? In which jobs? 

(d) considering the core set of skills required in different jobs, what possible 
career moves are there for jobseekers? Which jobs, although different, require 
a similar set of skills?’ 


Cedefop has set up a pan-European system for gathering and analysing 
information from online vacancies across all EU countries: ‘The classification of 
European skills, competences, qualification and occupations (ESCO) and complex 
big data analysis techniques were used to extract information on skills from the 
vacancies. The results are presented in the skills online vacancy analysis tool for 
Europe (skills-OVATE)’ (19). Skills-OVATE offers detailed information on jobs and 
skills employers demand in online job vacancies. The tool presents data collected 
from July 2018 until September 2020 in all EU Member States and in the United 
Kingdom. Skills-OVATE allows navigation of data based on the following 
categories: 

(a) online job vacancies providers; 


('4) For example, Ockham IPS used a web-spidering tool to map the labour market and 
desired qualifications and competences for cyber security professionals in the 
Netherlands: Ockham IPS/PLATO (2014). Arbeidsmarkt voor Cyber Security 
Professionals. 

(15) Top jobs refers in this quote to jobs most in demand. 


(8) http://www.cedefop.europa.eu/en/events-and-projects/projects/big-data-analysis- 
online-vacancies 
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(6) countries and occupations; 
(c) regions and occupations; 
(d) breakdown by occupation; 
(e) skills in occupations; 

(f) most requested skills; 

(g) skill sets in occupations; 
(h) sectors and occupations. 


Figure 7, for example, shows which skills are most often requested in online 
job vacancies for healthcare assistants in Ireland. 


Figure 7. | Most requested skills in online job vacancies for healthcare assistants 


in Ireland 
Online job vacancies by occupation (1 Online job vacancies by occupation (2 Online job vacancies by occupation (4- 
digit ISCO) digit ISCO) digit ISCO) 
Care workats 
Personal care workers 
OK IK K 
Most requested skills Treamap Minimum share of vacancss where skill is menboned 
(ESCO version 1) @ Word cious 22.00% I )100,00% 
provide information 
work as a team Communication 
assist customers adapt to change 
nursing principlesonline analytical processing person centred care 
work with nursing staff Irish communicate with nursing stafi 
use communication techniques provide customer follow-up 
adapt to changing situations 


NB: Data retrieval 6.2.2020. 
Source: Skills-OVATE. 


In general, such vacancy analyses provide comprehensive, detailed and 
timely insights into labour market trends, as well as evidence on employer skill 
demands. They enable new and emerging jobs and skills to be identified early. 
However, ‘using OJVs as a data source for labour market analysis has several 
limitations: 

(a) vacancies in some sectors and occupations are over-represented in OJV 
portals; 

(bo) the use of OJV portals differs across and within the countries, due to the digital 
divide and different employment structures; 
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(c) skills listed in a vacancy notice do not reflect the full job profile: employers 
tend to list only critical skills and qualifications to filter job applicants; 

(d) vacancy notices have to be machine-readable and use a standardised 
vocabulary and, given the quantity of data, some simplifying assumptions 
have to be made; 

(e) the same vacancy notice may be published on several websites and not 
necessarily correspond to an actual job opening’ (Cedefop, 2019b, pp. 9-10). 


Skills-OVATE does not provide a link to specific qualifications and currently 
only publishes skills related to ESCO. But, since qualifications are often mentioned 
in job advertisements (often serving as a proxy for skills), the feasibility for 
analysing such data and including the information on qualifications, and possibly 
also European qualifications framework (EQF) levels, could be explored ('’). 
However, vacancy analyses focus on the demand side only without taking the 
provision of education and training into account. Vacancy analyses therefore also 
do not provide data to complete the learning outcomes feedback loop. 


2.4. Forecasting approaches and Cedefop’s skills 
forecast 


The forecasting of skills demand and supply is seen as an important measure that 
can signal trends and complement other labour market information in order to 
adapt skills policies to highly dynamic and competitive labour markets. Countries 
have implemented national or regional forecasting models that use different 
methodologies and build on various data sources. 

Cedefop’s pan-European skills forecast is the only source of comparable data 
on future employment trends across Europe. It offers quantitative projections of 
future trends in employment by sector of economic activity and occupational group. 
It explores future job openings resulting from both new jobs and a need to replace 
existing workers. Future trends on the level of education of the population and the 
labour force are also estimated. Cedefop’s skills forecast uses harmonised 
international data and acommon methodological approach with the aim of offering 
cross-country comparisons about employment trends in sectors, occupations and 
qualifications. ‘The methodology uses a modular approach, with the following main 
elements: 

(a) the demand side (skill needs), focusing on employment (jobs); 


(17) On the Irish portal Job.ie, for example, the desired national qualifications framework 
(NQF) level of candidates’ qualifications is indicated — see: 
https://www.jobs.ie/ApplyForJob.aspx?ld=1918335. 


39 


Review and renewal of qualifications: 
towards methodologies for analysing and comparing learning outcomes 


(b) the supply side, focusing on available skills, the number of people 
economically active and the qualifications they hold; 

(c) imbalances, comparing the demand and supply side modules’ (Cedefop and 
Eurofound, 2018, p. 13). 


The individual modules are presented in Figure 8. 

This model provides perceptions on general trends but uses rather broad 
categories. For example, in relation to qualifications, the following three categories 
are used (Cedefop, 2012, p. 135): 

(a) low level (international standard classification of education (ISCED-97: 0-2); 
(6) medium level (upper and post-secondary (ISCED-97: 3-4); 
(c) high level (ISCED-97: 5-6). 


Like the other approaches presented in Chapter 2, it strongly focuses on the 
demand side and, while considering the supply side, it does not refer to the delivery 
of education and training. Thus, it is not possible to use the data for assessing the 
relevance of individual qualifications and their learning outcomes and _ for 
completing the feedback loop. 
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Figure 8. Modelling skill supply and demand 
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Source: Cedefop and Bear 2018, p. 15. 
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2.5. VET graduate tracking measures 


VET graduate tracking can be defined as the collection of quantitative micro and 
aggregate data and/or qualitative information about employment and social 
outcomes of people leaving VET. VET graduate tracking measures can provide 
more general information on VET graduates for certain qualifications (such as 
destination, employment status, salary level) and it might be possible to link these 
findings to more specific (types of) learning outcomes associated with certain 
qualifications. Information obtained in such measures can provide crucial 
intelligence on the quality of learning programmes and can be used to renew 
qualifications and programmes as well as to provide advice to prospective students 
(and their parents) on what career changes VET qualifications and programmes 
offer. The ability to track graduates is also considered a core component of 
effective quality assurance systems, as it provides a mechanism for gathering 
intelligence on skills utilisation in the labour market and placement rates. This is 
reflected in the European quality assurance framework for vocational education 
and training (EQAVET) which was established by a recommendation in 2009 ('°). 
It was further emphasised in other European policies, such as the 2015 Riga 
conclusions On a new set of medium-term deliverables in the field of VET for the 
period 2015-20 ('%), the Commission’s new skills agenda for Europe, adopted in 
2016 — that also emphasised the need for tracking VET graduates to gain a better 
understanding of their performance (2°) — and the 2017 Council recommendation 
on tracking graduates, which proposed making progress by 2020 on the 
establishment of graduate tracking systems (°'). 

In line with these policies, the European Commission has supported several 
initiatives to increase the knowledge of VET graduate tracking policies and 
practices in Europe, to enhance the use and quality of graduate tracking, as well 
as cross-country cooperation on this: 

(a) in 2017, the Directorate-General for Employment, Social Affairs and Inclusion 

(DG EMPL) published the study Mapping of VET graduate tracking measures 

at EU Member States (European Commission, 2017) which mapped national, 


(18) European Parliament and Council of the European Union, 2009; 
https://www.eqavet.eu/ 


('°) Latvian Presidency of the Council of the European Union and European Commission, 
2015: https://eu2015.lv/images/notikumi/VET_RigaConclusions_2015.pdf 


(°°) European Commission, 2016. 
(*') Council of the European Union, 2017. 
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regional, sectoral and institutional practices in VET and discussed scenarios 
for cooperation at EU level (7); 

(6) in 2018, the Commission established the expert group on graduate tracking 
(2018-20) which provides a forum for cooperation and mutual learning about 
graduate tracking and data analysis (73); 

(c) in 2019, the Directorate-General for Education, Youth, Sport and Culture (DG 
EAC) commissioned the study Mapping the state of graduate tracking policies 
and practices in the EU Member States and EEA countries (European 
Commission, 2020b). The study aimed to update and add to existing 
intelligence on graduate tracking mechanisms in higher education (HE) and 
VET to inform the work of the expert group on graduate tracking, ahead of the 
reporting required on progress towards achieving the recommendation in 
2020 (**). This study focused on national and regional measures. 


The 2017 and 2019 studies on graduate tracking measures collected 
information on the methodologies applied (such as the use of administrative data, 
surveys or a combination of both) and analysed the characteristics of the measures 
identified. Some results are presented in the following paragraphs. 

The list of indicators dealt with in both studies in the measures for tracking 
VET graduates shows that only rather general information can be collected using 
administrative data. Measures for tracking VET graduates based on administrative 
data alone are not sufficient to close the feedback loop. It is not a surprise that 
graduate tracking measures based on surveys can include more general indicators 
as well as more individual ones, such as satisfaction with training received or of 
employment acquired and relevance/utilisation of acquired skills at the workplace. 
In case of this last factor, the surveys analysed in the 2019 study refer to the 
following aspects (from the graduates’ perspective): 

(a) general matching of job and education (for example, adequacy of employment 
in relation to education: was the training or the certificate/qualification 
obtained a necessary requirement for the job?); 


(*2) The study was conducted by ICF consulting services limited (Belgium), in cooperation 
with 3s (Austria), under the framework contract EAC/47/2014 (request for services 
VT/2016/058). 


https://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail& 
groupID=3580&news=1 


(*4) The study was conducted by ICF consulting services limited (Belgium), in cooperation 
with 3s (Austria) and CHEPS (the Netherlands), under the framework contract 
EAC/47/2014 (request for services EAC/23/2019). 
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(b) matching of skills acquired during education and their utilisation in 
employment (for example, are the skills obtained useful in the current job, 
frequency of using the skills obtained in the current job, adequacy of skills 
level needed in the job and skills level obtained). 


The aims of VET graduate tracking measures are of interest: particularly 
relevant are those measures that can be used for the review and renewal of 
qualifications. The results of both studies show that measures to track VET 
graduates are mainly used for policy development and planning and less frequently 
for the review and renewal of qualifications or programmes at provider level. 

In order to be relevant for the completion of the feedback loop, the graduate 
tracking measures would have to relate to specific qualifications but this is not 
always the case. Around three quarters of the measures analysed in the 2017 
study can be linked to specific qualifications. The 2019 study did not consider this 
aspect but about half of the measures contain at least a link to level, subject and 
provider and not just to general categories (Such as ISCED level). 

With measures to track VET graduates based on administrative learning and 
employment data, it is only possible to collect ‘hard’ facts. Surveys must be used 
to investigate the satisfaction of graduates or their employers with the learning 
outcomes achieved, as well as the relevance and utilisation of acquired skills at 
the workplace. However, such categories are not included in all cases; when they 
are included, they are only treated from the perspective of the graduate and not 
from that the employer. On the basis of the two mapping studies, it is unclear to 
what extent individual learning outcomes or even types of learning outcomes have 
been taken into account in the measures analysed. It can be concluded that the 
review and renewal of qualifications and their learning outcomes are generally less 
often the focus of these measures. Therefore, the current tracking approaches for 
VET graduates in the EU Member States, despite considering the demand and 
supply side, are of limited relevance in terms of data collection to complete the 
feedback loop as understood in the context of this study. 


2.6. Employer reflection surveys 


Employer reflection surveys can be defined as approaches in which employers (or 
their representatives) are asked to provide their reflections on the relevance of 
qualifications in the labour market. These surveys are usually, in some form, part 
of renewal of qualifications processes. However, they are systematically 
conducted at a national level only to a limited extent. In this study, the research 
team identified the following examples in the 10 countries covered: 
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(a) surveys that address employer perception of and demand for qualifications; 

(b) employer reflection surveys without reference to learning outcomes included 
in qualifications; 

(c) employer reflection surveys with reference to learning outcomes included in 
qualifications. 


2.6.1. Surveys that address employer perception of and demand for 
qualifications 

The examples below include those studies and surveys that are very much based 

on employer consultation as a source of information, but do not take the shape of 

an employer reflection survey. 

In UK-England, several questionnaire-based surveys could be identified, that 
address employer perception of and demand for qualifications, usually within the 
context of skill demand. The Department for Education publishes a biennial skills 
survey, consulting employers about the abilities of the workforce and current and 
future workforce needs (DfE, 2017). The Office of Qualifications and Examinations 
Regulation (Ofqual, the independent qualifications regulator for England, 
responsible for regulating around 150 awarding organisations) has commissioned 
a survey of employer perceptions of vocational and technical qualifications (Pye 
Tait Consulting, 2018); the Edge Foundation (a civil society organisation 
concerned with vocational education) publishes a quarterly skills shortage report 
(*5). An interesting feature of the Edge Foundation bulletin is that it includes 
summaries of other reports: the DfE skill survey (see above), the employer 
perceptions survey (see above), the quarterly British Chambers of Commerce 
economic survey, the CBl/Pearson education and skills survey (10 published so 
far) and the quarterly Open University business barometer. These are all 
questionnaire-based surveys and their main relevance to this project is that they 
address employer perception of and demand for qualifications, usually in the 
context of skill demand. None of them have surveys (or data on surveys) of 
employer satisfaction with, or perception of, particular qualifications; they do not 
deal with the design features of qualifications, such as whether they are outcomes- 
based, nor is there any detail on specific qualifications or what employers think of 
them. They do, however, offer plenty of data on the amount and levels of training, 
understanding of the vocational qualification and VET system, employers’ 
perceptions and use of qualifications in general, and some data on skill utilisation 
(Edge Foundation). Of interest for this project could be the combination of methods 
used: 


(°°) For example Edge Foundation, 2019. 
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(a) the DfE employer skills survey used a sample of the population of businesses 
with more than two employees in England, Scotland and Wales. The core 
sample was interviewed, and a follow up questionnaire was issued to a 
subsample, with a response rate of 43%; 

(ob) the Ofqual-commissioned employer perceptions survey used an_ initial 
approach questionnaire, with follow-up interviews with employers, training 
providers and learners. 


Also, the French public employment service (P6/e emploi) publishes a number 
of relevant studies in this regard. The How do employers select the applicants they 
recruit? (Chamkhi et al., 2018) study, for instance, is based on a qualitative and 
quantitative employer survey that mainly includes questions regarding the last 
batch of recruitments (expected competences in the trade job, difficulties felt when 
trying to recruit, criteria used in the end to decide on whom to recruit), as well as 
one question about the parts of the CV that were closely reviewed. This type of 
survey is carried out regularly by Pdéle emploi, the most recent iteration of this 
survey has been issued in 2016 (Lainé, 2016). The statistical service of the Ministry 
of Labour (Direction de |’ Animation de la Recherche, des Etudes et des 
Statistiques, DARES) published a similar survey in 2017 (Section 2.6.2). 

The Bulgarian public employment service (PES) conducted an employer 
survey on local short-term labour demand in February 2018 (6). The study is 
carried out twice per year and is obligatory under the Employment Promotion Act. 
lt provides information on discrepancies between skills supply and demand in 
specific sectors and occupations, but it does not include questions related to the 
satisfaction of employers with skills and competences of VET graduates. 

The Danish Ministry of Education operates grant schemes that allow trade 
committees to obtain funds to commission research projects, in order to back up 
their work and provide input for the drafting of occupational profiles. They address 
specific sectors or themes, and often involve employers as sources of information, 
but not in the style of a reflection survey (?’). 

Also, in Spain, several studies have been identified that collect information on 
the demands and training needs of companies. Private employment agencies 
usually publish reports or general surveys in an unsystematic way, which gather 
opinions from employers on the difficulties in finding suitable professional profiles 
or on their competence requirements for the professional profiles. In addition, some 


(7°) The study is available in Bulgarian at: 
https://www.az.government.bg/pages/prouchvane-potrebnosti-rabotna-sila022018 
(Bulgarian Ministry of Labour and Social Policy, 2018). 


(?’) Danish Ministry of Children and Education, 2018a: 
https://www.uvm.dk/puljer-udbud-og-prisuddelinger/puljer/puljeoversigt 
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foundations — such as Funcas (Chamber of Commerce) — have undertaken 
qualitative approaches to collecting business opinions, in which they include some 
questions about the education system and what it offers. 


2.6.2. Employer reflection surveys without reference to learning outcomes 
in qualifications 

The following examples again use employers as a source of information and ask 

them about their satisfaction with the VET system and (in some cases) the 

satisfaction with specific VET qualifications. They do not refer directly to the 

content of the qualifications. 

In Denmark, regular ERS are carried out in order to ascertain the level of 
cooperation or cohesion between vocational schools and placement enterprises, 
and to ensure the wellbeing of learners in order to avoid students dropping out (7°). 
These employer reflection surveys are, however, not related to the development 
of qualifications in terms of learning outcomes. 

In Finland (7°), VET providers, as part of their obligatory quality assurance 
measures required by VET legislation, have their own feedback systems and, as 
part of these, there are also measures which gather information on the satisfaction 
with VET provision and the skills and competences of the graduates. How this is 
then done varies from one VET provider to another, but it tends to remain at a more 
general level, meaning that specific employer satisfaction with the skills and 
competences of VET graduates does not tend to form part of these feedback 
inquiries. 

In France, the interprofessional training centre for the food industry (CIFCA, 
centre interprofessionnel de formation des commerces de |’ alimentation) has 
published surveys on employer satisfaction, alternating with training quality (2015- 
16) (°°). The survey is based on a multiple-choice questionnaire and the target 
group includes enterprises with up to 10 employees. The questions address issues 


(78) Danish Ministry of Children and Education, 2018b: 
https://www.uvm.dk/erhvervsuddannelser/skoleudvikling/klare- 
maal/virksomhedstilfredshedsmaalinger 


(7°) In addition, the new Vocational Training Act and the reform provide for the renewal of 
the financing system for VET in Finland. Within the framework of the reform, the 
financing of VET will be renewed to the extent that a more emphasised part of the 
performance-oriented and efficiency-based financing elements will be introduced. 
These elements also include indicators of employer satisfaction. How these indicators 
are to be defined, monitored and measured has not yet been defined, but the system 
should be in place by 2022. 


(3°) CIFCA, n.d.: 
https://www.cifca.fr/cifca_images/audit_satisfaction_employeurs_cifca.pdf 
https://www.ifopca.fr/ifopca_images/audit_satisfaction_employeurs_ifopca.pdf 
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such as: the recruitment process, the training itself, the satisfaction with the learner 
during the on-the-job practice, whether the interviewee would hire the learner, and 
whether s/he would recommend the CIFCA training centre. Another example from 
France is the survey on job advertisement and recruitment (offre d’emploi et 
recruitment, Ofer) that was carried out by the statistical service of the Ministry of 
Labour in 2016 and focused on enterprises that recruited at least one employee 
between September and November 2015 (DARES, 2017). This is the only French 
national survey describing all the steps of recruitment, from the formulation of the 
need in terms of human resources to the satisfaction of the employer in relation to 
the recruitment. The themes addressed, and the questions asked, take into 
account the evolution of the labour market (for example, development of 
information and communication technologies, trade-offs by employers regarding 
the different types of contracts). 

In the Netherlands, Panteia carried out an employer satisfaction survey that 
focused on the distribution of graduates from all education levels, across (groups 
of) professions within the logistics sector. An interesting element in this survey is 
the inclusion of a section on the occupational context relevant to this sector, 
through a so-called PESTLE analysis (°"). 


2.6.3. Employer reflection surveys with reference to learning outcomes 
included in qualifications 

These surveys are aimed at employers, to measure their satisfaction with VET 
graduates working in their company in terms of the skills and competences they 
have acquired and the use of these skills in the workplace. The research across 
the 10 Member States covered by this study reveals that such ERS are not 
commonly used to complete the feedback loop on learning outcomes achieved. 
Other approaches are more often applied to ensure the relevance of learning 
outcomes of qualifications. In the European countries covered by this study, we 
only identified six examples of surveys that have employers reflect on the achieved 
learning outcomes as intended by the qualification. A detailed analysis of these 
examples is presented and discussed in Chapter 3. 


(3!) PESTLE or PESTEL analysis (formerly PEST analysis) refers to ‘a framework or tool 
used to analyse and monitor the macro-environmental factors that may have a 
profound impact on an organisation's performance’ (B2U, 2016, p. 1). 
https://www.business-to-you.com/scanning-the-environment-pestel-analysis/ 
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2.7. Closing the learning outcomes feedback loop: an 
assessment of different approaches 


Countries have different mechanisms for reflecting on the match between the 
supply of VET programmes and qualifications and labour market demand. A first 
type, often informal, is that VET providers have direct contact with employers, for 
instance related to work placements, internship or apprenticeships. Quality 
assurance mechanisms may also include references to tracking graduates and 
employer satisfaction. This allows VET providers and employers to discuss 
whether the content of the qualifications and the delivery of the programme are in 
line with labour market needs. Second, the supply-demand matching mechanism 
can be integrated into the VET governance structure. Labour market stakeholders 
are (partly) responsible for the VET system and therefore systematically reflect on 
the content of VET programmes. This can be institutionalised, for example in sector 
skills councils or trade committees. Third, specific research approaches can be 
applied to gather information on the relevance of VET qualifications and the 
content of VET programmes. The potential contribution of these research 
approaches to completing the feedback loop is briefly assessed below. 

VET graduate tracking surveys are a particularly valuable source for tracking 
the further development of graduates after completing their VET programme; they 
can provide an indication of whether the qualification is in demand on the labour 
market, and sometimes even which skills and competences have contributed to 
career development. However, many of these surveys and studies focus on 
general issues, such as whether graduates find a job easily or whether they were 
satisfied with the training received. The extent to which these surveys and analyses 
are able to capture the demand for specific learning outcomes and the orientation 
and profile of qualifications in detail is not yet fully explored. Moreover, graduate 
surveys are usually not able to capture a perspective on whether the intended 
learning outcomes are actually achieved. 

The other approaches explored provide important data for creating skills 
intelligence but are not sufficient for completing the feedback loop based on 
learning outcomes: 

(a) skill mismatch analyses and the European skills and jobs survey provide 
information on the degree of match between skills supply and demand, but 
usually at a higher aggregated level without reference to individual 
qualifications; 

(6) online vacancy advertisements, including Cedefop’s skills-OVATE, can be a 
valuable source of information on what kind of skills and competences are in 
demand in a particular occupation. However, this does not indicate whether 
or not these skills are provided in the education programmes that prepare for 
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this occupation. Moreover, it can be questioned whether it is at all possible to 
link vacancy data to specific qualifications and whether this can provide 
relevant information on graduates' skills presence; 

(c) forecasting procedures at national level and Cedefop's pan-European skills 
forecast provide insights into general future trends and do not typically relate 
to the perception of the learning outcomes achieved and realised by 
graduates. Further, the European skills forecast does not refer to individual 
qualifications but uses broad categories. 


Another observation is that existing approaches to identifying skills 
match/mismatch are strongly biased towards the demands of the labour market 
and often do not consider the supply side, the provision of education and training. 
VET graduate tracking and employer surveys can be exceptions of this. 

Figure 9 presents the positioning of the approaches and tools analysed related 
to their focus: demand or supply side on the one hand and individual qualifications 
and learning outcomes or broader categories on the other hand. 


Figure 9. +Positioning the approaches and tools analysed for closing the learning 
outcomes feedback loop 
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Source: Cedefop. 


It can be concluded that a more promising approach for completing the 
feedback loop seems to be the use of employer surveys which can ask whether 
employers are generally satisfied with graduates or whether they experience 
shortages, or what is the demand for specific qualifications in the labour market. 
Employer surveys exploring their satisfaction with their employees’ learning 
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outcomes are the most direct means of tracking the link between intended and 
actual or realised learning outcomes, as they ask employers whether recent 
graduates can actually apply the skills and competences that were ‘promised’ in 
their qualification. The use of such employer reflection surveys will be discussed 
in more detail in Chapter 3. 
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3.1. Introduction 


As indicated in Chapter 2, there are different ways to complete the feedback loop. 
Most existing approaches pay little attention to the content and profile of 
qualifications and operate at a more general level, making it difficult to use data to 
support the review and renewal of qualifications with a view to improving their 
quality and relevance. One approach that is not systematically implemented in 
European Member States, that might have the potential to provide the reflection 
on individual learning outcomes, is the employer reflection survey that takes into 
account the content of qualifications, as briefly introduced in Section 2.6.3. These 
surveys are aimed at employers, to measure their satisfaction with VET graduates 
working in their company, in terms of the skills and competences they have 
acquired and the use of these skills in the workplace. This study is particularly 
interested in surveys that ask questions about the intended and/or achieved 
learning outcomes of qualifications; or that provide information about which 
skills/competences are most (and least) relevant to the job of graduates and the 
extent to which the qualification has adequately prepared graduates for them. This 
excludes the more general surveys, such as those aimed at simply giving feedback 
on how well graduates from a particular institution or programme are received (in 
terms of how good they are at doing the job), without specifying the required and 
desired skillsets or aspects related to identifying skills mismatches. Only a few of 
the 10 countries surveyed have employer reflection surveys that look at the content 
of qualifications. In some countries this was due to the absence of any surveys 
addressed to employers; in cases where employers were addressed, the type of 
survey generally tended to focus on identifying skill needs rather than on gaining 
insights into their satisfaction with learning outcomes. In Chapter 3 we will further 
explore the sample of identified surveys, enriched by non-European approaches. 

In the following we focus on the ERS which aim at identifying the match 
between the intentions of the VET system (intended learning outcomes) and the 
needs of the labour market. Reflection by employers on whether VET graduates 
have actually achieved the intended learning outcomes, and whether the learning 
outcomes achieved and put into practice actually meet the needs at work, would 
complete the feedback loop at the level of learning outcomes. Our focus will be on 
aspects of design and methodology of the (six) ERS examples identified by the 
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national researchers in the European countries covered by this study. In addition 

to the surveys identified, the research team also considered another non-European 

survey (from Australia), which represents an interesting and well-developed large- 
scale ERS approach. The six European and the one non-European ERS, which 
are the focus of Section 3.1, concern the following: 

(a) Australia: 2018 employer satisfaction survey (ESS). The ESS is an annual 
national survey involving two rounds of data collection each year (in 
November and May). The ESS looks at university and non-university higher 
education graduates. The specific targets are the supervisors of recent 
graduates. The ESS in Australia is a national approach, with well-developed 
methodology. The downside of this ESS is that it focused on higher education 
and that it is a non-European survey; 

(o) Austria: employer survey from the University of Applied Sciences, Carinthia, 
2014 (°). This is a regional survey, as part of a one-off study carried out in 
2014 by the University of Applied Science (°9); 

(c) Ireland: Irish national employer survey (NES). This (national) survey covers 
graduates from higher education (*) and further education and training (FET, 
the lrish equivalent of VET), and distinguishes between higher education and 
FET in the analysis. Drivers for the NES are the national strategies for FET 
and higher education, both of which stress the need for closer relationships 
between business and education and training (**); 

(d) Lithuania 1, quality assurance (QA): survey on employers’ satisfaction with 
the skills and competences of VET graduates in the framework of the project 
European programme of the VET quality assurance national guidance points 
activities No 2016-0783/001-001 (°°); 


(3?) In national language: Arbeitgeberbefragung der Fachhochschule Karnten 2014 (UAS 
Carinthia, 2014). This survey does not refer to IVET, but to higher education. 
Nevertheless, it has been included in this analysis as there is no employer satisfaction 
survey (ESS) for IVET in Austria and the regional approach applied was considered 


interesting for the design of the ESS prototype in this study. 


In preparation for the institutional evaluation (in 2015) by the Austrian quality 
assurance body for higher education (AQ Austria). 


Higher education in Ireland includes institutes of technology, which provide, inter alia, 
higher VET programmes. 


An objective of the further education and training strategy 2014-19 is to ensure that 
the relevant FET provision is informed directly by employers. 


Implemented by the Lithuanian Centre for the Development of Qualifications and 
Vocational Training (2018). In national language: Darbdaviy pasitenkinimo profesinio 
mokymo jstaigy absolventy gebéjimais tyrimas, jgyvendinant ‘Europos profesinio 
mokymo kokybés uZztikrinimo nacionaliniy orientaciniy. punkty 2016 m. veiklos 
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(e) Lithuania 2, skills, competences and qualifications (SCQ): survey on 
employers’ satisfaction with the skills and competences of VET graduates 
executed by the Lithuanian Employers Confederation (LEC) in the framework 
of Erasmus+ mobility projects since 2016 (°”). This survey was developed and 
used by the employers’ organisation, following the needs and focused on the 
requirements of employers; 

(f) Netherlands, employer satisfaction measurement: transition of upper 
secondary vocational education (middelbaar beroepsonderwijs, MBO) to the 
labour market (°°). This is a survey developed by the Foundation for 
Cooperation on Vocational Education, Training and Labour Market (Stichting 
Samenwerking Beroepsonderwijs Bedrijfsleven, SBB) at the request of the 
Ministry of Education, Culture and Science (OCW), following a motion filed by 
Member of Parliament Straus (°%) in 2015. In this motion, she requested the 
inclusion of a national indicator on employer satisfaction into the government 
budget. The survey was conducted by an organisation that already has a 
registry of recognised (VET) training companies and the sample was 
expanded for the second iteration (including unregistered non-training 
companies); 

(g) Spain: employment monitor training needs in the Community of Madrid 
(Community of Madrid, 2016) (4°). This concerns a primary investigation of the 
market needs of workplaces through a survey of companies, representative in 
terms of economic activity and company size. 


In most of the examples studied, the ERS is either systematically performed 
(Australia, Ireland, Netherlands) or was developed as a one-off study (Lithuania 1 
and 2, Austria). Spain is an exception here, in that the survey was meant to be 
systematically performed (every two years), but was not repeated after the first 
iteration. Overall, it was indicated by national experts that ERS are either seen as 
not providing enough information within their country context to be used 
systematically, or employers are already involved in other, overarching feedback 


programa Nr. 2016 — 0783/001-001', vykdyta Lietuvos Kvalifikacijy ir profesinio 
mokymo plétros centro. 


(°”) In national language: Lietuvos versio darbdaviy konfederacijos atliekamas darbdaviy 
pasitenkinimo profesinio mokymo jstaigy absolventy, dalyvaujanciy Erasmus+ mainy 
programoje kompetencijomis ir kvalifikacijomis tyrimas (LEC, n.d.). 


(38) In national language: Meting tevredenheid werkgevers: Aansluiting MBO — 
arbeidsmarkt (SBB, 2018). 


(3°) A member of the Dutch House of Representatives at the time. 


(*°) In national language: monitor empleo: necesidades de formacién en la Comunidad de 
Madrid. 
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mechanisms for (VET) education so further surveys would only lead to increased 
administrative burden for employers. 
Sections 3.2-3.7 analyse and discuss the following design features of these 


a) the types of organisation involved; 

the main objectives of the ERS; 

the scope (in terms of geography, education level, economic sectors and 
specified occupations/qualifications); 

(d) the target respondents; 

(e) the general approach; 

(f) the content of the surveys in relation to learning outcomes. 


3.2. Types of organisation involved 


For the seven examples studied, it was found that the surveys are most often 
initiated by government authorities, though they are generally conducted by a 
secondary partner such as a research institute (as seen in Spain) or tripartite party 
(as seen in the Netherlands). The Australian ESS is commissioned by the 
Australian Government Department of Education and Training; it was originally 
developed and pilot tested by the Workplace Research Centre (WRC) at the 
University of Sydney and currently conducted by the Social Research Centre. One 
example from Lithuania is an exception (Lithuania-1, QA). In this case, the 
government authority is also the one conducting the survey. For two examples 
where a non-government authority was the initiator, the same organisation 
conducts the survey (Lithuania 2 (SCQ) and Austria). Table 4 provides an overview 
of the parties involved for the selected examples, as well as their roles in terms of 
initiating and coordinating/conducting the survey. 

In the Austrian case, the ERS was initiated by a (higher) VET provider, one of 
the universities of applied sciences, the UAS Carinthia. Government organisations 
involved are generally education departments (Australia, Ireland, Netherlands). 
Only Spain is an exception in this regard: the government authority initiating the 
ERS is an autonomous community (i.e. with its own parliamentary system), one of 
17 across Spain. When the survey is conducted by a party other than the initiator 
(Australia, Ireland, Netherlands, Spain), this is generally done by research 
institutes. The exception is the Netherlands, which is the only example of a tripartite 
organisation being involved in either roles. 
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Table 4. Overview of initiators and coordinators/conductors for the six 
examples selected 


ESS 


example Initiator(s) Coordinator(s)/conductor(s) 
1. AT VET (HE) provider: VET (HE) provider: 
? University of Applied Sciences (UAS) University of Applied Sciences 

Government authority: Research institute (private company): 

2. AU Australian Government Department of The Social Research Centre owned by the 
Education and Training Australian National University (ANU) 

we Research institute 
3.ES Government authority: (private company): 


Comunidad de Madrid Simple Légica Investigacién SA 


Government authorities (joint project): 
the Higher Education Authority (HEA), 
which led the project (*"); 

4. 1E the Irish further education and skills service 
(SOLAS), responsible for FET (“); 
The Department of Education and Skills; 
Quality and Qualifications Ireland (*°) 


Research institute 
(private company): 
Fitzpatrick associates economic consultants 


Government authority: Government authority: 
5.LT1 The Centre for the Development of The Centre for the Development of 
‘ Qualifications and Vocational Education Qualifications and Vocational Education and 
and Training of Lithuania Training of Lithuania 


Empl r iation: a 
ployel-assoclalle Employer association: 


6.LT2 puSinieSs Employers Confederation of Business Employers Confederation of Lithuania 
Lithuania 
om Tripartite organisation (“*): 
7. NL Government authority: Foundation for Cooperation on Vocational 


Ministry of Education, Culture and Science Education, Training and Labour Market (SBB) 


Source: Cedefop based on country templates. 


(*1) HEA is the national agency with responsibility for the effective governance and 
regulation of higher institutions and the higher education system. It is accountable to 
the Minister for Education and Skills, and its board includes representatives from 
higher education and business but no trades unions). HEA drove the 2013 and 2015 
surveys; VET content was increased for the 2018 survey. 


(42) SOLAS is an agency of the Department of Education and Skills and is governed by a 
board, made up of representatives from VET providers, the third sector, and 
businesses (no trades unions, currently). 


(8) An independent State agency, responsible for promoting quality and accountability in 
education and training services, whose board comprises people from a range of 
backgrounds, including the Union of Students in Ireland. 


(**) Collaboration between government, employer and worker representatives. 
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3.3. Main objective of an employer reflection survey 


In examining the main objectives of the selected examples, an initial observation 
is the extent to which they relate to the scope of this study: the relevance of skills; 
focus on VET graduates; and involvement of employers). Most of the examples 
(four) can be linked directly to VET graduates. In the Netherlands, the survey can 
be linked to VET graduates at EQF levels 1 to 4, while Ireland is linked to both VET 
and higher education graduates. The narrowest target group is seen in Lithuania, 
where both examples focus on a subset of VET graduates: the survey related to 
quality experience (LT 1) is aimed at all VET graduates (at EQF levels 3-4), while 
the survey related to international mobility (LT 2) is only for those involved in 
Erasmus+ projects aiming at (learning) mobility between 2017 and 2018 (Social 
Research Centre, 2019, p. 6). 

In the Austrian case the objective to establish and develop good contacts with 
the main employers in the region was expressed; this could be seen as a 
secondary goal for the prototype to be developed within the scope of this study. It 
is also as a potential (positive) side effect of systematically performing an ERS in 
general, especially in terms of involving employers who feel unheard (not already 
included in existing feedback mechanisms, etc.). Table 5 presents an overview of 
the main objectives of the ERS examples that were selected. 


Table 5. Main objectives of ERS, by country 


VET a We) 0) (=Xeq (V=%-9 


e — Establish and develop good contacts with the main employers in the region 
1. AT e = Obtain information which could be of interest for the future development of different 
study programmes and curricula 


To measure, help monitor and better understand: 
e — The specific skills and attributes employers need in their business 


° How well higher education is equipping graduates for the workforce 
2. AU 


The objective explicitly refers to ‘employer views of the technical skills, generic 
skills and work readiness of recent graduates provide assurance about the quality 
of Australia’s higher education sector’ (Social Research Centre, 2019, p. 6). 


To determine: 


3. ES ° Mismatches in the qualification of the personnel (internal or external) of the 
. companies based in the Community of Madrid 
e Training needs for employed and unemployed workers 


Ascertain employers’ views on: 
e Overall quality of recent higher education and FET graduates 
° Barriers to graduate recruitment 
e Collaboration between business and education institutions 


4.1E 
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VET a We) 0) (=Xeq (AV=X-9 


To evaluate: 
4.LT1 e The quality of the VET provision, as a part of development of the national VET 
: quality assurance programme 
e The employers’ satisfaction in skills and competences of graduates 


To evaluate: 
e — The contribution of the international mobility of the VET graduates to their 
5.LT 2 competence development 


e The satisfaction of employers with the skills and competences of recently employed 
graduates, before and after the Erasmus+ mobility (4) 


To measure: 


6. NL e The satisfaction of employers with the match between the education (VET) 
. programmes (*°) and the labour market needs, based on data from training 
companies hiring recent VET graduates (e.g. MBO). 


Source: Cedefop based on country templates. 


In terms of the extent to which these examples refer to learning outcomes, 
Table 5 shows that these are not explicitly mentioned in the main objectives of any 
of the selected examples, although they are implied in the two examples from 
Lithuania as these aim at evaluating the satisfaction of employers with the skills 
and competences of graduates (LT 1 and LT 2) and in the example from Australia. 


3.4. Survey scope 


In the ERS examples studied, the scope in terms of geography was generally either 
regional (Spain, Austria) or national (Australia, Lithuania, Netherlands). Ireland 
differed in that the ERS is conducted at national level but also distinguishes 
between Dublin and outside Dublin. In Austria, the survey is conducted in only one 
specific region, Carinthia, while in Spain, the survey is aimed at one of its 17 
autonomous communities, the Community of Madrid. Lithuania (ESS 1 and 2), the 
Netherlands and Australia are national surveys. 

In terms of education levels covered, the most specific ERS examples were 
those from Lithuania, where the surveys are aimed at (employers of) graduates at 
EQF levels 3 and 4. The least specific example was the Spanish case, which aims 
at graduates of all EQF levels. The Dutch ERS refers to all VET graduates, 
corresponding to EQF levels 1 to 4, while the examples from Australia, Austria and 


(45) The newly employed VET graduates were offered mobility for the on-the-job training 
in the other EU countries. 


(*°) More specifically, the MBO levels 1-4, which correspond to the Netherlands 
qualifications framework (NLQF) and EQF levels 1-4. 


58 


CHAPTER 3. 
Employer reflection on achieved learning outcomes: surveys examined 


Ireland are aimed at higher EQF levels: Austria: 6 and 7; Ireland: 5 to 8; Australia 
at higher education. 

For economic sectors, most of the ERS analysed either cover a specified 
range of NACE codes (Spain, Ireland), or they are aimed at a specific set of sectors 
(Austria, Lithuania 2, Netherlands) which may overlap between examples (using 
the same or at least similar sectors). The exception is one of the examples from 
Lithuania (LT 1) which is not specific to economic sectors and instead covers all 
economic sectors and areas. The Irish ERS was structured to be representative 
across NACE codes (reporting at NACE level 1), while the Spanish ERS covers all 
two-digit 2009 national classification of economic activities (CNAE) sectors, 
corresponding to the statistical classification of economic activities in the European 
Community (nomenclature statistique des activités @conomiques dans la 
Communauté européenne, NACE level 2). Table 6 provides an overview of the 
sectors as specified for the ERS examples in Lithuania (LT 2), Netherlands and 
Austria. 


Table 6. Overview of economic sectors identified in ERS examples 


e = Agriculture 

e Construction 

e Engineering industry and ICT 
e Healthcare 

e Services to persons 

e Transport and logistics 


LT 2 6 sectors 


e Business services 

e Food, green and hospitality 

e Healthcare, wellness and sports 

e ICT and creative industries 

e Mobility, transport, logistics and maritime 
e Technique and built environment 

e ©6Trade 

e = Specialistic craftsmanship 


NL 8 sector chambers (47) 


e Building, construction, architecture 
e Business and management 

e Engineering and IT 

e Health and social care 


AT 4 sectors 


Source: Cedefop based on country templates. 


(47) The company conducting the survey (SBB) has categorised every existing study 
programme into one of eight sector chambers. 
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In terms of the focus on specific occupations, only two of the ERS examples 
covered a specific set: those in Lithuania (LT 2) and Austria. In the Austrian ERS, 
a set of highly skilled occupations in the respective (four) sectors are specified, 
and the Lithuanian ERS (2) focuses on 22 selected professions within their 
respective set of (six) sectors (*°). Table 7 presents an overview of the selected 
professions as seen in the ERS examples for Lithuania and Austria. 


Table 7. Occupations specified in ERS (LT 2, AT), by corresponding economic 


sector 

ESS Economic Occupations 

example | sector 

Agriculture Agriculture worker 
Construction Plasterer, road builder 
Enainearin Welder, fitter, CNC operator, ICT network administrator, java 
in ce 9 software designer, adjuster of hardware, internet designer, 
and ier electronic technician for energy sector, equipment 

LT2 maintenance worker, mechatronic of automated systems 
Healthcare Healthcare assistant, cosmetologist 
Services to Hairdresser, cook, waiter, SME business assistant, social care 
persons worker, producer of publicity measures 
oe and Logistics operator 
ogistics 
Building, F Among others: 
construction, 


: architects, planners, building and construction managers, etc. 
architecture 


Among others: 


Business and public management specialists, tourism managers, managers 
management for SMEs, business and administration specialists, marketing 
specialists, etc. 
AT Among others: 
Engineering and high-tech manufacturers, electrical engineers, electronics 
IT specialists, IT specialists, hardware and software specialists, 
etc. 


Among others (*): 

Health and social social workers, streetworkers, nurses and health care 

care managers, dietologists, radiologists, midwives, logopaedics, 
ergotherapists, biomedical analytics, etc. 


NB: (*) This does not include medical doctors. 
CNC stands for computer numerical control. 


Source: Cedefop based on country templates. 


Even though there is considerable overlap in the sectors, occupations 
included in the Austrian example can be considered higher positions compared to 
Lithuania. This is to be expected, considering the differences in the EQF levels the 
examples cover: EQF levels 6 and 7 in Austria, as opposed to levels 3 and 4 in 
Lithuania. 


(48) These occupations were surveyed in 2017. The list of surveyed occupations may 
change and expand each year. 
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Overall, it seems that the more specific or detailed the distinction — in terms of 
economic sectors and professions within them — the more difficult it becomes to 
find similarities between examples (and between countries) due to differences in 
phrasing. Having to match categories manually, based on their context or the 
development of overarching terms to group categories under (through post 
coding), would require an increase in workload. It would also depend on the person 
performing the manual matching or post coding, allowing for interpretation bias that 
could be avoided by using a more commonly known set of broad definitions (or 
defining them specifically for the survey) such as the NACE classification (at the 
lowest level of detail, L1) or occupational groups. 


3.5. Targeted respondents 


Except for Lithuania — which targets multiple parties (including providers, 
graduates, employers and teachers) — all ERS examples analysed were found to 
target employers as the main survey respondents. The Australian example, 
however, draws its sample from respondents from a prior graduate survey 
(graduate outcomes survey, GOS), generally without an explicit demarcation in 
terms of inclusion or exclusion criteria (such as a minimum company size). Only 
the ERS examples identified in Australia, Ireland and the Netherlands have 
specified a demarcation for the graduates to be assessed: 

(a) in lreland, for the purposes of the ERS survey, a graduate was defined as 
someone who has been recruited within the past 24 months, and who has 
also completed their higher education or FET qualification within the past 24 
months; 

(b) in the Netherlands, the graduate has to have been in employment at the 
company for at least six months and needs to have obtained their qualification 
within the past two years to be considered a recent graduate. Additionally, if 
several recent graduates have been in employment for at least six months, 
the employer is to assess the one that was most recently hired; 

(c) in Australia, the ERS included workplace supervisors of higher education 
graduates (not self-employed or working in a family business) who completed 
the GOS). 


These demarcations are similar in so far as they both define that a (recent) 
graduate should have obtained their qualification in the past two years, but are 
different in how recently the graduates should have been hired by the employer. 
The Irish example has the broadest demarcation in this regard, as including 
graduates hired within the past 24 months would also include those hired as recent 
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as a few weeks; for the Netherlands, a minimum employment period is included in 
the demarcation (of at least six months). Considering that it generally takes time 
for an employer (or manager/supervisor) to get a good sense of the graduate’s 
skills overall (and for graduates to showcase them at their new job), it would be 
better to include a minimum requirement on employment period in the prototype. 
Doing the same for how recently the qualification was obtained, however, could 
exclude graduates that were employed almost immediately after graduation (or 
already employed before) and would not be recommended for the scope of this 
study. Table 8 provides an overview of the demarcations used within this context 
(if at all) for the selected ERS examples. 


Table 8. Overview of target respondent demarcations and descriptions 
identified in the ERS examples 


ESS : = 


No specific demarcation, as respondents Key employers in the region had been identified before the 
AT were already registered in a database interviews took place. However, the notion of key employers 
maintained by UAS Carinthia. has not been defined more precisely. 
AU No specific demarcation for identification of The only demarcation is that the graduate is not self-employed 
possible respondents. or working in family business 


Overall, the survey targets companies of the Community of 
No, the survey refers to all current company Madrid, with two or more workers, in the most representative 
ES personnel, and the future hiring branches of activity (e.g. with more than 5 000 workers). It also 
expectations for new employees. includes a survey for senior managers and personnel training 
managers (“°). 


For the purposes of the survey, a graduate was defined as 
IE Not for respondents, only in terms of someone who has been recruited within the past 24 months, 
defining graduate. and who has also completed a higher education or FET 
qualification within the past 24 months. 


There are four survey questionnaires proposed for the VET 
providers. These survey questionnaires are designed for the 
VET graduates, VET students, VET teachers and employers. 


LT1 No specific demarcation for identification of 
possible respondents. 


Both employers and graduates are surveyed, by providing the 
LT2 No specific demarcation for identification of description of competences for self-evaluation and for 
possible respondents. evaluation of the progress of graduates in their competence 
development. 


(*°) Which asks them about the occupational structure of their company, current, past and 
expected number of workers per job, retirement and replacement, active hiring of 
workers, training needs for their current staff and for candidates who are in the labour 
market, and organisational changes that may be affecting the respondent’s activity 
and that of their staff. 
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ESS , cae 


For this survey, the recent graduate refers to those who have 
been in employment there for at least six months and have 
Yes, for both companies and graduates. obtained their qualification within the past two years. An 
addition to the demarcation is that If multiple recent graduates 
Companies were selected from the registry have been working for at least six months, the employer is to 
maintained by the conducting party (SBB), assess the one that was most recently hired. 
specifying those companies that currently 
have one or more recent VET graduates in Further, it is important to note that — for the 2016 iteration — 


NL employment. only recognised training companies were included in the target 
Companies were excluded from the sample —_ population, while the 2017 iteration also included other 
if they: organisations (i.e. non-training companies and/or unrecognised 
- did not have a valid email address for the training companies). Since there were no significant differences 
VET-training contact person; found between recognised and non-recognised organisations 
- were not located in the Netherlands. (and the increase in administrative costs for obtaining the 


supplementary data from the Central Bureau of Statistics, 
CBS), future iterations of this survey are expected to only 
include recognised companies in the target population. 


Source: Cedefop based on country templates. 


Regardless of the fact that most examples were found not to have specified a 
demarcation for the target respondents, it is possible to deduce some (less implicit 
demarcations), since most of the ERS examples indicate having done some 
preselection (sampling) beforehand: 

(a) in Austria, the UAS Carinthia identified key employers in the region from a 
database of employers for graduates and students of UAS Carinthia study 
programmes; in some cases these were specific key persons within the 
organisation that were already known to UAS Carinthia; 

(bo) in Ireland, the survey sample was drawn from an existing database of 
companies by setting quota controls on several factors before using random 
selection of companies; 

(c) in Spain, even though no specific demarcation was defined, the sample was 
drawn from a database of companies of the Community of Madrid, including 
only those with two or more workers in the most representative branches of 
activity (for example those with more than a total of 5 000 workers across 
companies); 

(d) in the Netherlands, the SBB drew the sample from their registry of recognised 
training companies, on the condition of currently having one or more recent 
VET graduates in employment; they further specified that the graduate has to 
have been employed there for at least six months, and has obtained their 
qualification within the past two years. 


It is important to note that most of the examples indicate the use of an existing 
database or registry, of either learners or employers. One other approach identified 
is using the full sample for a selected project/programme, as seen in one of the 
ERS examples in Lithuania 2. In this case, the sample included the enterprises 
belonging to the employers’ organisation in Lithuania and those in the other 
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countries that accepted graduates for the Erasmus+ mobility traineeship organised 
by the employers’ organisation. 

From the demarcations and sampling approaches discussed in this section, 
we can distinguish between two aspects or types of demarcations, within the scope 
of this study: 

(a) graduate demarcations (as seen in Ireland, Netherlands), such as: 

e — min/max length of current employment (°°); 

e how recently the qualification was obtained; 

e whether to consider one specific graduate or all recently hired graduates 

that fit the demarcation (i.e. overall satisfaction score); 
(6) employer/company demarcations (Spain, Netherlands), such as: 

e company size (for example minimum number of employees); 

e —_sector(s) they are active in; 

e region(s) they are active in; 

e whether they have hired a recent (VET) graduate. 


3.6. General survey approach 


Approaches identified in the ERS examples selected are similar in that they directly 
contact the potential respondents, either by email or by telephone, with one 
exception: for the Lithuanian quality assurance ERS (Lithuania 1), respondents 
were contacted by the providers. In terms of the methods used for conducting the 
survey itself, it was found that, generally speaking, either digital surveys were used 
or (Semi-structured) interviews; only Lithuania allowed respondents to choose 
between a paper questionnaire or completing the survey online. To provide 
additional insights, the approach in terms of identifying and contacting 
respondents, and in terms of how (and how often) the survey is to be conducted, 
is discussed below for each example. 

In Austria, employers were identified through the VET provider, in this case 
also the conductor of the survey. The University of Applied Sciences Carinthia 
(Fachhochschule Karnten) maintains a database of employers for graduates and 
students during their entire practical phase. They approached directly those who, 
in their view, were the ‘most important’, to schedule face-to-face interviews, of 
which a total of 40 were conducted (from an initial sample of 60). In some cases, 


(°°) A maximum could be used to prevent misleading results, for example when graduates 
that are employed for longer periods are assessed with higher satisfaction scores due 
to them having improved their skills through performing their job (i.e. by putting them 
into practice) instead of being correlated/linked to the quality or relevance of the skills 
as they were taught during their education. 
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specific key persons within the employer organisations were already known to UAS 
Carinthia. The interviews lasted between 30 to 60 minutes each and were based 
on guidelines, as well as a questionnaire, developed by those responsible for the 
study programmes (or heads of faculties. In terms of timeline and frequency, this 
example is considered a one-off study, conducted in 2014 in preparation for 
institutional evaluation by the Austrian Quality Assurance body for Higher 
Education (AQ Austria), which took place in 2015. 

In Ireland, employers were identified through the Bill Moss and Associates’ 
database of companies. Quota controls were generated for company size, sector 
and region to ensure representation across NACE codes (at level 1) — based on 
the most recently available Central Statistics Office (CSO) statistics for businesses 
in lreland — and then a sample of 8 936 companies was randomly selected. Some 
6 986 of these leads were contacted, with a final sample of 760 companies being 
surveyed, in line with the targeted sample of 750 employer organisations (5'). A 
computer-assisted telephone interviewing (CATI) telephone survey was conducted 
between June and July in 2018, with a total of 181 companies indicating they had 
hired FET graduates (47 in Dublin, 134 outside Dublin). The CATI approach was 
introduced for the 2018 survey partly to ensure the SME base was properly 
represented, unlike in the previous surveys where companies completed the 
surveys themselves. The 2018 survey (published in January 2019) is the third 
iteration, following previous surveys in 2013 and 2015. However, since the random 
sampling approach was introduced in 2018, there is a break in the timing and it 
has not yet been decided when the next survey will be carried out (°). 

In Spain, employers were identified through stratified random selection from 
the DUAE database (2013 directory of local units of economic activity), supplied 
by the Directorate-General for employment strategy and development with a total 
of 65 535 companies. After sample selection, the contact information of employers 
was obtained through the same registry and the survey was conducted through 
computer-assisted telephone interviewing, with each interview taking a global 
average of 12.12 minutes (10.59 minutes in companies with two to nine workers 
and 38.90 minutes for those over 250). These interviews were carried out by a 
team of simple logic surveyors, with proven experience in conducting business 
interviews, trained in the specific aspects of this work. In terms of timeline, the 


(5') Of the 6 986 organisations contacted, 2 180 declined to complete the survey, 1 768 
registered an engaged dial tone or went straight to voicemail, 1 587 requested that the 
interviewer called back at another time, 497 did not fit any of the quotas required, 194 
were no longer trading and 760 completed the survey. 


(52) This is due to the fact the next iteration is not included in the 2019 SOLAS work plan 
and will be decided by SOLAS and the HEA. 
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initial expectation was to conduct the survey every two years, but it has not been 
carried out again since 2015. 

In Lithuania 1, potential respondents (graduates, their employers and their 
teachers) were identified and contacted through VET providers (who are also 
target respondents themselves) using their existing networks, or through students, 
graduates or other social partners. Four questionnaires were prepared to measure 
the satisfaction of these main target groups, allowing analysis of quality 
measurement instruments and good practices in Lithuanian general education, 
vocational training and higher education institutions. The employers were 
approached directly by the VET providers, and, for all target groups, the surveys 
could be executed through a printed questionnaire or by completing the 
questionnaire online, taking on average between 10 and 15 minutes. These 
surveys can be executed regularly (for example annually), as well as on specific 
occasions, such as self-evaluation of the activities implemented by VET providers 
before the external quality evaluation. 

In Lithuania 2, the Business Employers Confederation of Lithuania — which is 
also the coordinator of the corresponding Erasmus+ mobility projects — selects the 
employed VET graduates for Erasmus+ mobility training in other EU countries. The 
information about such graduates is received from the graduates themselves, who 
apply for the Erasmus+ mobility after graduation from VET school, as well as from 
the enterprises (members of confederation) which employ these graduates. This 
survey is only conducted among graduates that participated in Erasmus+ mobility 
organised by the employer confederation, surveying approximately 50 graduates 
each year. Frequency depends on the organisation of these projects, as the survey 
is executed immediately after the completion of each traineeship. Group 
discussions and consultations are organised after the exercises, where 
participants present their acquired experiences. 

In the Netherlands, employers were identified through the independent 
(tripartite) organisation conducting the survey (SBB), which maintains its own 
registry of recognised training companies. For the first iteration (2016), only 
recognised training companies were selected for inclusion in the sample. Potential 
respondents were contacted by email to determine whether they have hired recent 
VET graduates, and if so, to request their participation in the (online) survey. For 
the second iteration, however, regular organisations (non-training or 
unrecognised) were approached as well, following a request from the Ministry of 
OCW. For the expansion of the sample, the SBB cooperated with the Central 
Bureau of Statistics: this organisation maintains the (national) general 
organisations registry (Algemeen Bedrijven Register) and also keeps track of 
which organisations have hired recent (VET) graduates. The Central Bureau of 
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Statistics (Centraal Bureau voor de Statistiek, CBS) was responsible for the 
sampling, fieldwork and data preparations in 2017, while SBB performed the 
analyses and drafted the report. Potential respondents received a letter from CBS, 
requesting them to participate through an online survey. The letter contained a 
login code and (unique) user number to allow access to the online survey, which 
is estimated to take between 5 to 15 minutes; non-respondents were reminded 
with another two letters. The survey is to be conducted once every two years and 
future iterations will follow the set-up of the first iteration. Given that the analysis 
report showed that differences between recognised and non-training companies 
were so minimal (if any at all), the next iteration will, again, only draw its sample 
from recognised training companies (without the increased costs from 
collaborating with CBS). 

In Australia, the general approach is to work together with the higher 
education institutes to finalise the questionnaires and invite the graduates. They 
are asked to provide contact details of the employers as well. At the end of the 
graduate outcomes survey, all graduates in employment (but not self-employed or 
working in a family business) were asked to provide details (name, email and/or 
phone number) of their current supervisor, so that the supervisor could be invited 
to take part in the ESS. Promoting the ESS involved reaching out to both graduates 
and employers through institutions and various key organisations. A number of 
peak bodies — including the Australian Association of Graduate Employers 
(AAGE), the Australian Institute for Teaching and School Leadership (AITSL) and 
the National Association of Graduate Careers Advisory Services (NAGCAS) — 
were also made aware of the ESS through conferences and meetings (Social 
Research Centre, 2019, p. 11). Dual methodologies were utilised in the 2018 ESS, 
with online and CATI workflows established to support supervisor participation. 
The online workflow was activated as the primary workflow for all records with a 
valid email address. If a supervisor did not respond to the email invitation or 
reminders, they were then entered into the CATI follow-up workflow where a phone 
number had been provided. As described in the national report, the population of 
the ESS consisted of 95 121 domestic and international graduates, who responded 
in the 2018 GOS that they were employed. Of these, 10 216 employed graduates 
provided sufficient contact details to approach supervisors, yielding a supervisor 
referral rate of 10.7%. From direct supervisors, a total of 5311 valid survey 
responses were collected across all study levels, representing a supervisor 
response rate of 52% (QILT, 2019, p. vi). 

None of the selected examples were found to include specific incentives to 
stimulate participation by respondents (no financial incentives or obligated 
participation). The surveys and studies identified show several channels through 
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which the contact information of the target group can be obtained (as well as for 
contacting them), for both graduates and employers: 

(a) through education institutions; 

(b) through employers; 

(c) through employed graduates; 

(d) through government administrative offices (such as citizen’s administration). 


Another channel, however, has not been used in the ERS examples selected: 
the option to publish an open invitation on digital platforms. These may include 
newsfeeds for students/graduates maintained by education institutions or those 
aimed at employees of companies, or social media (LinkedIn, Facebook and other 
news channels). Depending on the design of the desired survey, the available 
methods of survey completion (paper, online, phone) and the target groups 
identified, several channels can be combined in order to maximise exposure and 
increase response rates. 


3.7. Content of the questionnaires 


Few of the examples studied had a specific focus on learning outcomes, while 
most surveys do include a measure/set of concepts similar to learning outcomes. 
None includes a way to assess the proficiency levels related to the learning 
outcomes (vertical dimension). To provide additional insights, the content of the 
questionnaire in terms of the intended unit of analysis was examined: who/what is 
assessed (which is generally not the respondent him/herself), how and which 
learning outcomes are included, and, if the vertical dimension of learning outcomes 
was addressed. The following was found in terms of the survey content. 

In Austria, the survey aims to gain insight into the satisfaction of employers 
with the competences of graduates from UAS study programmes, specifically 
those at UAS Carinthia. It also seeks information on what kind of competences will 
develop within the coming years, so that the UAS can redevelop study 
programmes according to future labour market needs (differentiated in bachelor 
and master qualification levels). The survey should also help to reflect the 
distinction of competences acquired in bachelor and master courses from 
employers. The academic distinction was seen as not fitting into employers’ human 
resources (HR) decisions. In terms of analysis, a distinction is made between 
bachelor and master graduates and between labour market sectors (for example 
economic sectors) related to the main study fields. Learning outcomes are only 
presented in open questions and are based on an existing classification (by 
Erpenbeck and von Rosenstiel, 2003), which distinguishes between five categories 
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of competences. The vertical dimension of learning outcomes is not directly 
addressed in this survey. 

In Ireland, the survey aims to provide insights into employers’ views on overall 
quality of (FET and higher education) graduates, as well as their satisfaction levels 
across a range of attributes. The vertical dimension is not addressed in this survey. 
Learning outcomes are presented as a range of workplace attributes that 
employers are asked to provide their satisfaction with, broadly related to learning 
outcomes. A set of personal attributes is also presented. These lists of attributes 
were mainly elaborated from areas of concern, as expressed by employers, in 
terms of the skills they perceived to be poorly developed among graduates. 

The survey in Spain is not actually linked to specific graduates, but rather to 
qualifications or groups of qualifications (by sector, EQF level or another grouping 
factor). The analysis focuses on training modules (which are similar to learning 
outcomes), by asking an open question about the type of training required by 
employers for each type of occupation; the responses are then codified following 
similarity with existing training modules, as seen in the national catalogue of 
professional qualifications (catalogo nacional de cualificaciones profesionales, 
CNCP). Training modules can be compared with sets of learning outcomes, but 
this does not allow for addressing the vertical dimension (hierarchy of complexity). 

In Lithuania 1, the survey aims to evaluate the effectiveness of VET 
programmes — in accordance with the Erasmus programme on VET quality 
assurance national guidance point activities — by providing insights into 
employment of graduates and their careers after graduation. In terms of learning 
outcomes, it follows a typology of work/occupational specific and key skills. The 
vertical dimension of learning outcomes is not addressed in this survey. 

In Lithuania 2, the aim is also to evaluate the effectiveness of VET 
programmes — more specifically the Erasmus+ mobility traineeship — by assessing 
the achievements of the VET students and graduates during the practical training 
and work executed during the mobility visit abroad. In terms of learning outcomes, 
the competences are described following the vocational qualifications transfer 
system (VQTS) competence matrix (5°), which distinguishes between areas of 
competence development and the steps of competence development which 
consist of partial competences. These competences are arranged in this matrix 
following the didactic principle (for example from simple to more complex), so it 
gives some insight into the vertical dimension of the learning outcomes. No 
distinction is made between knowledge and skills. 


(°°) The VQTS model was also used as one of the reference points in WA1, see Luomi- 
Messerer, 2009. 
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In the Netherlands, even though the survey is aimed at employers, the 
questions asked and information retrieved are for the recent graduate that the 
employer selected to assess. If multiple recent graduates have been working for 
at least six months, the employer is to assess the one that was most recently hired. 
In terms of learning outcomes, this survey includes a list of seven keywords or 
short statements, representing relevant skills or tasks that an employee can 
perform. Future iterations will include a separate question for digital literacy, 
distinguishing between four aspects. It is important to note that the keywords/short 
statements do not come from an existing typology: the list was originally developed 
for an unpublished pilot study on the appreciation of VET education among 
employers. This survey does not address the vertical dimension of learning 
outcomes. 

The Australian ESS consists of the following modules: 


(a) module A: introduction and screening; 

(b) module B: overall graduate preparation; 

(c) module C: graduate attributes scale, employer (GAS-E); 
(d) module E: institution specific items; 

(e) module F: close. 


The ESS does not refer to specific learning outcomes of qualifications. 
Instead, it gathers information on the overall graduate preparation in relation to the 
qualification of the selected graduate (5). Supervisors can also provide information 
on the main ways that the qualification prepared the graduate for employment and 
in what ways it could have done this better. Essentially, this line of questioning 
allows for gathering information on the intended (and achieved and observed) skills 
of a wide range of qualifications in their respective occupations, as the supervisors 
are asked to consider only one specific graduate. Further, the ESS has developed 
their own skills classification (in 2017) which distinguishes between five 
overarching types of graduate skills. Supervisors are presented with sets of 
attributes and skills for each of these five types and asked to indicate the extent to 
which they agree that the specific skill or attribute has prepared the graduate for 
the job. 

In terms of the way in which learning outcomes are presented — within a 
context of using typologies of learning outcomes (skills and competences) — all 


(54) This includes questions on: whether the obtained qualification is a formal requirement 
for the graduate’s current job (Y/N); to what extent it is important for the graduate to 
have this qualification or a similar one to do their job well (five-point scale, ‘not at all’ 
to ‘very important’); and whether they would hire someone with the same qualification 
for a relevant vacancy. 
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ERS examples indicated using either an existing or developed typology. Table 9 
provides an overview of the typologies used for each respective ERS example. 


Table 9. Overview of typologies identified in ERS examples 
ts} rea 9 ater 2 
example Existing typology? Description/categories 

e Professional know-how 

oe ieee of categories of # Tpeme tan orucomprtenes 
AT competences, as designed by Erpenbeck . Problom-solvirig competence 
and Rosenstiel, 2003 e Ability to work in a team 

e Ability to communicate 

Five overarching skill types: 

e Foundation skills: regarding general literacy, 
numeracy and communication skills and the 
ability to investigate and integrate 
knowledge 

e Adaptive skills and attributes: regarding the 

y, ability to adapt and apply skills/knowledge 
-- and work independent! 
Following the graduate attributes scale, é “Teamwork cee sicis: 
AU te sare a ed oul regarding teamwork and interpersonal skills 
attributes or skills for each skill type e Technical and professional skills: regarding 
the application of professional and technical 
knowledge and standards 

e Employability and enterprise skills: 
regarding the ability to perform and innovate 
in the workplace 
(*) 

This resulted in the following typology for the 
training needs: 
Somewhat e Technical formation 
The open-answered training needs are e Personal or managerial skills 
ES codified — following their similarity with e Computing: tools or programmes 


existing training modules, as seen in 
national catalogue of professional 
qualifications 
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New or emerging issues 

Unimportant training needs (i.e. those 
indicated less than five times across 
respondents) 
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ats) apa my rar ; 
example Existing typology? DY-X-Yod ao (Co) ayer: 1 (=e (o) gd (=\-) 


(a) Workplace attributes: 
e Computer and technical literacy 
e Working effectively with others 
e Effective verbal communication 
e Numeracy/processing/interpreting data 
e Effective written communication 
e Working effectively on their own 
e Application of technical knowledge 
e Attention to detail 


Yes e Commercial awareness 
IE Classification of workplace attributes and e Entrepreneurial skills 
personal attributes ¢ Foreign language capability 


(b) Personal attributes: 

e Professionalism and work ethic 

e Reliability 

e Positive attitude 

e Ethically and socially aware 

e Personal commitment 

e Adaptability and flexibility 
Ability to cope with work pressure 


Overview of specific and key 
workplace/occupation skills: 
e Application of theoretical knowledge in 
practice 
e Application of practical skills 
e Respecting work safety requirements at 
work 
e ICT skills and ICT literacy 
e Abilities to work with documents 


Somewhat e Abilities to learn to work with new 
LT 1 The survey follows a typology of specific techniques and technologies 
and key workplace/occupation skills e Communication skills (with customers and 
colleagues) 


e Following quality requirements at work 
e Foreign language skills 

e Ability to work in team 

e Ability to work independently 

e Responsibility and fair attitude to work 
e Positive attitude to work 

e Working culture 

e Other (please indicate) 


In this matrix, for each separate 
qualification/profession, areas of competence 
development are defined and split into the 


Yes necessary steps of competence development, as 
LT2 Following the method of describing them __ well as the partial competences they consist of. 
in a VQTS competence matrix Last, the steps of competence development and 


their partial competences are arranged by 
following the didactic principle — from simple to 
more complex. 


72 


CHAPTER 3. 
Employer reflection on achieved learning outcomes: surveys examined 


ERS nen 9 rae : 
example Existing typology? Description/categories 


List of seven keywords/short statements: 
e Professional knowledge 
Occupation-specific skills 
Professional attitude 
Language and math skills 
Relationship with colleagues and 
supervisors 
e Willingness to put in effort for the company 


Somewhat 
Following a set of keywords or short 
statements on the relevant skills or tasks 


an employee can perform, developed for Capacity for self-development and to further 
NL an unpublished pilot study. educate themselves in their field/profession 

Additionally, a separate question was ae 

developed for future iterations of the Four aspects of digital literacy: 

survey, which describes four aspects of e ICT base skills: efficient use of soft- and 

digital literacy (digitale geletterdheid). hardware 

e Media literacy: conscious, critical and active 
use of media 


e Computational thinking: using ICT-tools for 
problem-solving 

e Information skills: efficient collecting and 
processing of information using ICT 


(*) The full overview of attributes and skills per overarching skill type is presented in Table 10. 
Source: Cedefop based on country templates. 


As Table 9 shows, there is great variation in the way(s) learning outcomes 
(skills and competences) are part of the ERS identified. It is difficult to identify 
specific learning outcomes, as they are usually not referred to as such in these 
surveys; instead, they are implicitly embedded in the typologies and skills lists 
surveyed. Spain is the only example using an existing resource (catalogue of 
training modules) to link training needs to. It is these training modules that could 
be linked to learning outcomes, to some extent. Some of the other examples were 
found to use a relatively broad set of existing or specifically defined categories 
(Netherlands, Austria) for skills and competences. Ireland also uses a specific set 
of categories, though the main difference here is that it distinguishes between 
workplace and personal attributes — two overarching categories, along with a set 
of skills for each — which are more specifically phrased compared to those used in 
the Netherlands and in Austria. 

Another key finding from the comparison of the content of the questionnaires 
and the typologies used is that there is a dominance of transversal skills and/or 
learning outcomes over occupation-specific skills. When referring to the latter, the 
typologies include broad categories, such as using professional knowledge or 
applying occupation-specific skills, without going into the detail of what this entails 
in specific occupational contexts. This reflects the fact that transversal skills can 
be surveyed more broadly across different occupations, while occupation-related 
skills are only applicable for one specific occupation; this makes it challenging to 
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develop a survey approach that can balance between broad applicability and 
tailoring to specific occupations. It could, however, also relate to weak links 
between VET qualifications and occupations, causing the surveys to focus more 
on transversal learning outcomes that are visible regardless of occupation. This 
imbalance in skills included in the typologies, highlights a significant challenge for 
ERS in general: the balance between generality and specificity. 

An interesting categorisation in terms of being both broadly applicable to a 
wide set of qualifications, and including a high level of granularity, can be found in 
the Australian example. This developed its own typology for the survey, called the 
graduate attributes scale, that distinguishes between five overarching skill types 
for graduates, and specifies a set of attributes for each type. The full overview of 
this typology is presented in Table 10. 


Table 10. ESS graduate attributes scale (GAS-E) 


Skills type Attributes and skills 


Eight categories: 
e = Oral communication skills 
e Written communication skills 
e ~=Working with numbers 
e = Ability to develop relevant knowledge 
e Ability to develop relevant skills 
e Ability to solve problems 
e = Ability to integrate knowledge 
e Ability to think independently about problems 


Foundation skills 


Six categories: 
e Broad background knowledge 
Adaptive skills e Ability to develop innovative ideas 
and attributes e Ability to identify new opportunities 
e Ability to adapt knowledge to different contexts 
e Ability to apply skills in different contexts 
e Capacity to work independently 


Five categories: 
e = Working well in a team 
Teamwork and e Getting on well with others in the workplace 
interpersonal skills e Working collaboratively with colleagues to complete tasks 
e Understanding different points of view 
e Ability to interact with co-workers from different or 
multicultural backgrounds 
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Skills type Attributes and skills 


Six categories: 
e Applying professional knowledge to job tasks 
Technical and e Using technology effectively 
professional skills e Applying technical skills in the workplace 
e Maintaining professional standards 
e Observing ethical standards 
e Using research skills to gather evidence 


Seven categories: 
e Ability to work under pressure 
e Capacity to be flexible in the workplace 
e Ability to meet deadlines 
e Understanding the nature of your business or organisation 
e Demonstrating leadership skills 
e Demonstrating management skills 
e Taking responsibility for personal professional development 


Employability and 
enterprise skills 


Source: Social Research Centre, 2019. 


This typology has a reasonable number of overarching categories, as well as 
an average of six specified attributes for each overarching skill type. The terms 
used for the skills types and attributes are not tailored to specific sectors or 
occupations, meaning they can still serve as relatively broad/general descriptions. 
In the balance between generality and specificity, the Australian example provides 
a bit more detail on the specific occupation-related skills, while still keeping it 
applicable to multiple occupational contexts (cluster: technical and professional 
skills). 

There is also great variation between ERS examples in terms of how 
questions regarding learning outcomes were phrased in the survey, as well as the 
way(s) in which respondents were expected to answer. Except for Spain, which 
used open text fields to identify specific training needs, most examples used their 
typologies or classification as a set of scales; these seem to overlap to some 
extent, although they vary in their answer categories. Table 11 provides an 
overview of the questions that were within the scope of our study and the scales 
or answer categories that were used; these support the prototype designed for this 
study (in Chapter 4). 
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Table 11. 


toh) 
example 


AT 


AU 


LT 1 


LT 2 


Overview of questions and scales/answer categories related to 
assessing skills and competences (broadly linked to learning 


outcomes) 


ba oN A= Lae me [UC=x-y Colarmecxer-lcelialem(cr-laal tae] 


outcomes phrased to assess satisfaction with 
them? 


What are the most important competences that bachelor 
graduates need to have in order to be well matched to 
your sector/industry? E.g.: 

(a) professional know-how; 

b) implementation competence; 

c) problem-solving competence; 

d) ability to work in a team; 

e) ability to communicate. 


aS Se 


Supervisors are presented with sets of attributes and 
skills for each of these five overarching skill types (as 
defined in GAS-E typology) and were asked to indicate 
the extent to which they agree that the specific skill or 
attribute has prepared the graduate for the job. 


(1) Please rate your level of satisfaction with your further 
education graduate recruits as they relate to the following 
workplace attributes. 

(2) Please rate your level of satisfaction with your further 
education graduate recruits as they relate to the following 
personal attributes. 


Evaluate the readiness of our students, who perform the 
practical training at your enterprise for independent work, 
according to the aspects and ranking provided, from 1 
(very poor) to 5 (very good) (do not consider separate 
persons, but the totality of students). 


The survey questionnaire consists of a list of partial 
competences, specifically selected for the qualification in 
question. It uses a structured format (VQTS matrix) 

which distinguishes between steps of competence 
development. Each partial competence is then assessed 
by the VET student/graduate him/herself, as well as by 
the representative ‘employer’ (HR manager, supervisor of 
practical training). 
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Which scale(s) or answer 
categories were used? 


N/A 


The scale is scored as: 

1 = strongly disagree 

2 = disagree 

3 = neither disagree nor agree 
4=agree 

5 = strongly agree 

6 = not applicable 


The scale is scored as: 

1 = All satisfactory 

2 = 75% satisfactory 

3 = 50% satisfactory 

4 = 25% satisfactory 

5 = None satisfactory 

6 = Don’t know/not applicable 


The scale is scored as: 
1 = Very poor 

2 = Bad 

3 = Satisfactory 

4 = Good 

5 = Very good 

6 = Not applicable 


The scale is scored as: 

1 = unable to perform 

2 = can perform only with the help 
of more experienced specialist 

3 = can perform independently by 
using provided written or oral 
instructions 

4 = can perform independently 
with some supervision 

5 = can perform completely 
independently by attaining 
required level of quality 
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Co) Arla me L(x) (Col at-mactel-lcel laren (=x-laaliare] 


ESS 3 A E Which scale(s) or answer 
example pees phrased to assess satisfaction with categories were used? 
The scale is scored as: 
The list of skills/learning outcomes is presented asaset ‘1 = (very) bad 
NL of statements, where the employer can indicate the 2 = bad 
extent to which they are satisfied with the level of skill of 3 = not good, not bad 
the graduate being assessed on a five-point scale. 4 = good 
5 = very good 


Source: Cedefop based on country templates. 


Table 11 shows that employers are generally either asked directly for their 
reflection on a list of skills/statements or more tailored questions (lreland, 
Netherlands), or through specifying the readiness of the graduate or ability to 
perform the skills or attributes in their job (Lithuania 1 and 2). Within the scope of 
this study, however, the phrasing seen in Austria is closest; it directly asks for the 
relevance of each (competence) category for graduates in terms of most important 
for them to be well-prepared for performing the job. 

As well as the examples shown in Table 11, some additional questions were 
used in the Australian ESS that may be of interest within the scope of this study. 
This includes questions on: 

(a) whether the obtained qualification is a formal requirement for the graduate’s 

current job (Y/N); 

(bo) to what extent it is important for the graduate to have this qualification or a 
similar one to do their job well (five-point scale: not at all to very important); 

(c) whether they would hire someone with the same qualification for a relevant 
vacancy. 


3.8. Challenges associated with employer reflection 
surveys 


The national experts highlighted a number of challenges in designing ERS to 
capture the relevance of learning outcomes in their national contexts. 


The content of qualifications is too heterogeneous to be assessed in a 
generic ERS tool 

Applying ERS for mapping the relevance of learning outcomes can raise issues of 
the difference between national and regional contents of qualifications. In Ireland, 
while expected learning outcomes are specified at national level, these represent 
an approximation of the learning outcomes within programmes. A total of 16 
education and training boards (geographically based) devise programmes, 
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drawing on nationally defined qualification components, and they are increasingly 
being encouraged to set minimum intended learning outcomes that fit with needs 
within their areas (rather than copy/pasting them from the national specifications, 
as typically happens at present). So, programmes need to reflect the national 
expected learning outcomes, but they may be drafted differently. 

In some countries there might be optional parts in qualifications, meaning that 
the set of learning outcomes on which employers will have to reflect can differ per 
individual graduate (qualification holder). In Ireland, the national specifications of 
qualifications contain a significant number of optional components; this may give 
rise to much variation at local level in the actual experience of learners in what 
appears to be the same qualification (in terms of its title). Similarly, in the 
Netherlands, an emerging number of elective parts (keuzedelen) are provided, 
ranging from general education subjects (for instance, another modern foreign 
language) to specific technical skills training (such as operating drones), making it 
difficult to collect employers’ reflections on a fixed set of learning outcomes. 


Qualifications have weak links with occupations and the labour market is 
not accustomed to reflecting on the content of qualifications 

Other challenges relate to whether employers are accustomed to reflecting on 
learning outcome statements as included in qualifications. In some countries, such 
as Bulgaria, employers are only recently becoming more involved in discussions 
concerning the VET system and the VET qualifications; they are not yet familiar 
with the language of education. Even in countries with a longer tradition of 
employer involvement in VET, the quality of the reflections on learning outcomes 
depends on who are the respondents on the employers’ side. The respondents 
can be company instructors directly working with VET graduates, HR managers, 
or company directors. An adaptation of the language of the VET standards may be 
needed, meaning that the language of the survey (learning outcomes expressions) 
may need to be adapted, depending on what type(s) of employer representatives 
are involved. It is also questionable whether any supervisor of a graduate can 
oversee the whole set of acquired learning outcomes, including the non-occupation 
specific and more transversal ones. Answering in a team (within the company), 
however, would be difficult to implement practically. 

Another aspect here is the weak link between qualifications and occupations 
and jobs. Assessing the relevance of learning outcomes makes more sense when 
there are close links between qualifications and jobs, i.e. when a specific 
qualification leads to a specific specified job. This works best in more traditional 
sectors and in mature VET systems. In some countries, however, these conditions 
are not in place, leading to graduates finding jobs that are not directly linked to their 
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qualifications. The satisfaction of employers might, under these circumstances, not 
say much. 


The outcomes are compromised by high levels of subjectivity and low 
response rates; ERS can be costly 

Another set of challenges concern methodological issues, largely related to bias. 
The measurement of employer satisfaction is always subjective: judgement of 
whether a graduate possesses the learning outcomes strongly depends on the 
perspective of the respondent. For example, a very effective, hardworking line 
manager may find the new recruit not highly productive, and express general 
dissatisfaction, where the same recruit would be seen as fully satisfactory in 
another, less judgemental context. Further, employers might have different 
benchmarks for assessing their satisfaction. One might assess a graduate against 
expectations for a mature worker, another might assess against expectations for 
an intern. 

There are also methodological challenges over low response rates to 
employer surveys, due to lack of motivation to engage and attitude in answering 
questions in general. Employers often have difficulties in providing explicit and 
trustworthy information on VET programmes. Another problem with standardised 
surveys is the lack of flexibility to adjust to particular conditions of different sectors 
of economy or types of enterprises, which can also make some questions irrelevant 
to employers, or difficult to answer. 

ERS are considered costly and if they are designed to reduce costs, they will 
not be able to look at learning outcomes (particularly occupational ones) at the 
desired level of granularity. Related to this, there is generally either a lack of 
institutional and human resources in the economic sectors (sectoral 
organisations/councils) and social partner organisations or an uneven distribution 
of resources across sectors to support ERS. 

Given these challenges, any national picture of employer reflection at the level 
of learning outcomes would (increasingly) be an approximation. A more fruitful 
approach would be to work on lower levels of abstraction, launching small-scale 
tailored surveys by VET providers to the employers of recent graduates. This 
increases the relevance of the questions asked, while simultaneously increasing 
the chances of finding motivated employers, willing to respond. 
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3.9. Relevance and usefulness of the employer 
reflection survey 


Despite the identified challenges, five of the national experts stated that ERS could 
be useful in their country because of the opportunities they provide. This is the 
case for Bulgaria, France, Lithuania, the Netherlands and UK-England. In Bulgaria 
their value would lie in providing information on VET programmes and the whole 
VET system, but considerable resources would need to be invested in the 
development, implementation and evaluation of the results of such surveys to do 
this. In the UK-England, ERS could be an important feature in the quality 
assurance and review of qualifications; it is surprising that this is not the case in 
England, particularly among those who design and market the qualifications. It may 
that this is done on a commercially confidential basis, since qualifications are 
designed and marketed by awarding organisations which, despite their charitable 
status, compete for learners; even if they were to obtain feedback from employers 
on their qualifications through ERS, they would be unlikely to release them into the 
public domain. Some of these awarding organisations have their own research 
functions (°°) but there is little evidence of sector skills organisations conducting 
surveys of this kind (°°). The State regulatory body, Ofqual, does no more than 
carry out general perception surveys, which do not provide detailed intelligence on 
employer reflection with specific qualifications. However, it is unlikely that Ofqual 
could perform such an exercise comprehensively with the resources currently at 
its disposal. A further hindrance is the large number of qualifications that exist in 
the UK, even in the same occupational area with subtle but significant differences 
between them in terms of learning outcomes. There is an important gap here, 
which needs to be filled. The absence of strong sectoral employer bodies is a 
factor, as is also the possible commercial confidentiality of the information 
gathered. It is likely that such information would provide an important feedback 
loop between the labour and VET markets, and it would have the additional benefit 
of providing potential VET candidates with much-needed intelligence when making 
decisions about which qualifications to pursue. Given that considerable amounts 
of public money are involved in maintaining the VET market, it could be argued 
that this is an important government function not yet exercised as vigorously as it 
should be. 


(5) See, for example, City and Guilds’ research: 
https://www.cityandguildsgroup.com/research 


(5°) It should be noted that these are now privatised organisations run on a commercial 
basis and there may be little or no incentive for them to carry out such work. 
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In the Netherlands a systematic approach to mapping whether graduates are able 
to show acquired learning outcomes (as assessed by the employers), is lacking. It 
could be helpful — at the level of VET institutions — to assess this regularly, to see 
whether the education and training leads to sustainable outcomes. It could also be 
relevant at the national level, to inform the renewal process of qualification files, so 
ERS are used to improve employer engagement in the design of VET qualifications 
and governance of the VET system. It can also be applied in quality assurance at 
sectoral levels and inform the delivery of VET programmes at provider level. 

In some countries the national expert questioned whether such ERS would be 
at all useful or desirable in their national contexts. This was the case for Denmark, 
Ireland, Spain, Austria and Finland. Arguments include the fact that cooperation 
between VET schools and employers is already carefully monitored and evaluated 
(in Denmark), so an ERS does not provide added value. In Austria, ERS would not 
add value to the dual VET system, as the link between the content of qualifications 
and the labour market is already carefully monitored and VET providers (schools 
and companies) have to follow centralised curricula. The ERS are more relevant 
in higher level (higher education) vocational/professional education and training as 
higher education providers have many more opportunities to develop their own 
curricula, and do not have to follow centralised curricula. But even there — at least 
for UAS — governance provides for a crucial role for employers in the design and 
development of curricula; they must be part of a so-called development team 
appointed for the accreditation process. More generally, when comparing 
‘heartland’ VET (EQF levels 3 and 4) with VET at higher levels (as part of higher 
education), tripartite arrangements at these higher levels are less developed, so 
that higher education providers include less systematic input from labour market 
stakeholders into programme design. 

Given the strong links in place within the Member States and feedback 
arrangements between the labour market and the VET system, the introduction of 
regular ERS is questioned for producing any vital new input to VET. It may even 
be perceived as an unnecessary administrative burden by employers. The need to 
have ERS seems limited in systems where there are strong governance and 
system links between the labour market and the VET system, and where VET 
qualifications are governed and delivered in close cooperation with employers. It 
is likely that in more countries there are already (more direct) feedback loops within 
the system, to tailor the VET provision to emerging economic and societal needs. 
The ERS might be more relevant for Member States where the VET system is 
operating at relative arm’s length to the labour market. In these situations, it could 
provide a valuable contribution to closing the feedback loop, and in assuring the 
relevance of the learning outcomes provided within the VET qualifications. When 
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referring to the main types of feedback mechanism identified in IVET systems in 
Europe (Markowitsch and Hefler, 2018), the ERS might be more relevant in 
participatory and statist models compared to coordinated models and liberal 
models. This is in line with the findings of the Cedefop study The role of learning 
outcomes in supporting dialogue between the labour market and education and 
training: the case of vocational education and training (Auzinger et al., 2017). This 
study concluded that ‘learning outcomes descriptions are able to facilitate 
continuous dialogue, or multiple feedback loops throughout the entire process from 
developing/renewing qualifications to the delivery of VET programmes and finally 
the certification and the integration of graduates into the labour market. At all these 
stages, relevant stakeholders (employers, education providers) can use the 
learning outcome descriptions to inform the VET system of necessary changes’ 
(Auzinger et al., 2017, p. 130). Learning outcomes facilitate discussions within the 
process of delivering VET qualifications and make external feedback loops 
(through quantitative surveys) less relevant. 
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In Chapter 4, a specific approach to conducting employer reflection surveys is 
described and the outcomes of the pre-test are discussed. Before discussing the 
methodological approach, a general outline of the prototype is provided. 


4.1. Prototype general outline 


The main objective of the employer reflection survey approach is to support VET 
providers to match their offers better to the needs of their direct labour market 
stakeholders. As such, the scope of the ERS approach is institutional, as it is a 
reflection survey among the direct beneficiaries (graduates and employers) of the 
VET programmes offered/organised by VET providers. The ERS enables VET 
providers to engage with graduates and also with their employers to assess their 
reflection on the learning outcomes offered by the VET provider. At the institutional 
level, VET providers should be able to select and approach most recent graduates 
and usually already know the employers of graduates in their region. It is on this 
level that the ERS provides the most direct added value: it offers VET providers an 
opportunity to engage more with their beneficiaries (both graduates and 
employers) and reflect together with them on the outcomes of the programmes 
offered, also reflecting on the regional labour market they usually serve most. 
Hence, the ERS prototype is not an accountability tool but rather a tool to initiate 
dialogue between VET providers and employers (and graduates). However, the 
data collections should allow aggregation of the data to sectoral, national and 
cross-national levels. The main objectives of an ERS approach are presented 
schematically in Figure 10. 
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Figure 10. Objectives of an ERS approach 


The objectives of an ERS approach are: 


Comparing [4] 
sectoral data to compare different VET systems in 
and 0} how employers assess the achieved 
national data level learning outcomes 
(3) 
National- to generally see tendencies at national 
Analysing cross-sectoral level how employers value the VET 
provider-level data level system in providing specific sets of 
learning outcomes 


(2] 
to identify sectoral emerging needs in 
terms of learning outcomes 


(1] 
to support VET providers to better align 
their provision to the needs of their 
direct labour market stakeholders 


National-sectoral 
Clustering level 
provider-level data 


Provider level 
Data gathering 


Methodological condition: 
A skills reference point that is specific and meaningful at provider level, 
but allows aggregations to national/sectoral and cross-national/sectoral level 


Source: Cedefop. 


In terms of demarcation, the testing looked at two qualification profiles: one 
for healthcare assistant and one for ICT service technician. The testing took place 
in two countries, the Netherlands and Lithuania. The ERS was conducted with 
graduates and employers with the following characteristics: 

(a) VET graduates that: 

e had obtained their qualification in the last two years; 

e worked at least 16 hours per week in the related occupation; 

e had been employed in the current position for at least three months; 

(6) employer/supervisor of the VET graduate that: 
e had or has employed a graduate from the VET provider. 


In order to ensure individual responses are not identifiable and to obtain 
relevant data, the testing exercise aimed to obtain answers from at least five 
graduates and five employers for each VET provider and each qualification profile. 


4.2. Methodological approach to the pre-test 


The workflow of the ERS in the two countries was as follows: 

(a) step 1: the VET provider was selected to test the ERS prototype. The 
approach and content of the questionnaires was discussed with 
representatives of the VET provider. The VET provider put forward a 
characterisation of the VET qualification in terms of a learning outcomes 
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profile in relation to the reference point for the VET qualification (questionnaire 
for VET provider); 

step 2: the VET provider sent the link to the graduate questionnaire to the 
target respondents. In the invitation email, the researchers were also 
introduced. The researchers also asked the VET providers to send out 
reminders. Although the VET provider was responsible for inviting the 
graduates, individual data gathered were not available to the VET provider; 
step 3: the VET provider compiled a list of employers that potentially hired 
graduates from the programme and sent the invitation link to participate in the 
employer survey to the these. This survey also included questions on whether 
the employer consented to being contacted by the researchers to discuss the 
aggregated results; 

step 4: the researchers interviewed a number of employers that agreed to be 
approached when finalising the survey on their experience with the survey 
(length, level of detail, usefulness). 


The questionnaires were provided in the national languages (Lithuanian and 


Dutch), to increase response rates, and were tailored to the two selected 
qualification profiles. While the preferred method is to gather the data in digital 
form, the testing was also able to use paper-based questionnaires, which the 
researchers could distribute. 


The frequency and timing of an ERS depends on the request from VET 


providers. In a way, the same approach can be taken as in the Selfie methodology 
(Box 4), under the assumption that — in the future — certain aspects of the workflow 
can be automated. For the testing of the prototype, a one-off survey was 
conducted. 
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Box 4. Selfie (self-reflection on effective learning by fostering the use of 
innovative educational technologies) 


Selfie is a free, customisable tool to help schools assess where they stand with 
learning in the digital age. Selfie anonymously gathers the views of students, 
teachers and school leaders (and, in the case of work-based learning, also of 
employers) on how technology is used in the school. This is done through a survey 
which consists of short statements and questions and a simple one-to-five answer 
scale. While covering a broad range of topics, it should take around 20 minutes to 
complete the survey. 


The school can, through the online Selfie tool and dashboard, tailor the questionnaire 
to its needs. There are mandatory questions, optional questions and there is the 
possibility to add own questions. Through the tool, the school can invite the different 
stakeholder groups. The input gathered from these groups is generated into a school- 
specific Selfie report, highlighting the strengths and weaknesses of the school in 
applying digital tools. This report provides a basis for discussions with the different 
stakeholder groups on the different viewpoints and what could be improved in the 
school. 


Source: Selfie (°”). 


43. Askills typology for an employer reflection survey 


The ERS approach, in line with the above, has three questionnaires: 
(a) one questionnaire for VET providers; 

(b) one for graduates; 

(c) one for employers. 


In the questionnaires, when focusing on questions related to learning 
outcomes, it was agreed to use a skills typology that is broad enough — so that it 
can be used regardless of the sector, occupation or qualification in question — but 
still allows for respondents to score several specific items for each category 
included. In the preparation of the testing, the research team explored different 
approaches to constructing a suitable skills typology to be used. In this, the team 
looked at international reference points (Section 4.3.1) and existing national skills 
typologies (Section 4.3.2) before constructing an own tailored skills typology 
(Sections 4.3.3 and 4.3.4). 


4.3.1. Use of international reference points 
In search for a suitable reference point, first the reference points as discussed in 
previous parts of the overall study were considered: ESCO, the occupational 


(°”) https://ec.europa.eu/education/schools-go-digital/how-selfie-works_en 
https://ec.europa.eu/education/schools-go-digital/about-selfie_en 
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information network (O*NET), worldskills standards specifications (WSSS), VQTS. 
These reference points, however, include long lists of skills items, making it hard 
for graduates and employers to navigate. Further, while it is positive that these 
reference points are attuned to specific occupations, this is not an advantage for 
an ERS that is scalable. A suitable reference point for the ERS will have to include 
broader items that materialise differently for different occupations. Hence, the WA1 
reference points were not taken into further consideration to function as the ERS 
list of learning outcomes. However, there are developments related to further 
rationalising the knowledge, skills and competences (KSCs) in ESCO which are 
interesting to consider (°°). In rationalising the top-level ESCO skills and knowledge 
categories, a pattern was identified of categories of skills that are oriented towards 
communication and interaction and that gradually move towards the more 
instrumental equipment-oriented ones. This resulted in the following top-level 
classification (Table 12). 


Table 12. Draft ESCO skills and knowledge classification, revised 4 October 2019 


Revised level 1 categories 


New de New level 1 title Definition 
Communicating, collaborating, liaising and negotiating 
with other people, developing solutions to problems, 
Gamniunieaton creating plans or specifications for the design of 
S1 collaboration and abaativit objects and systems, composing text or music, 

y performing to entertain an audience, and imparting 
information, knowledge and skills, using written, oral, 
visual or electronic means 

Collecting, storing Conducting studies, investigations and tests, 
so monitoring arid using maintaining records; managing, evaluating, 
: we processing, analysing and monitoring information and 
information elite 
projecting outcomes 
a F Providing assistance, nurturing, care, service and 
$3 Peslotinig and casing 101 support to other people, and ensuring compliance with 


others Ae 
rules, standards, guidelines or laws 


Developing objectives and strategies, organising work 
activities, allocating and controlling resources and 
leading, motivating, recruiting and supervising people 
and teams 


Managing people, 
S4 activities, resources, and 
organisations 


(5°) Please note that the ESCO skills pillar has been and will be further developed and 
improved; this report takes into account the developments and proposals up to 
October/November 2019. 
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Revised level 1 categories 


4 ue ode New level 1 title Definition 
Using computers and other digital tools to develop, 
install and maintain ICT software and infrastructure 

S5 Interacting with computers —_ and to browse, search, filter, organise, store, retrieve, 
and analyse data, to collaborate and communicate 
with others, to create and edit new content 


Sorting, arranging, moving, transforming, fabricating 
and cleaning goods and materials by hand or using 
handheld tools and equipment. Tending plants, crops 
and animals 


S6 Handling and moving 


: Building, repairing, installing and finishing interior and 
oy Ponsiuenind exterior structures 

Controlling, operating and monitoring vehicles, 
stationary and mobile machinery and precision 
instrumentation and equipment. 


Working with machinery 


a and specialised equipment 


Source: Draft ESCO skills and knowledge classification, revised 4.10.2019, p. 2. 


In the context of revising the list of ESCO transversal skills and competences, 
developments can also be reported. Although still under debate, several 
considerations point in the direction of clearly separating three levels of skills and 
competences: 

(a) foundations — personal traits — values and attitudes; 
(6) transversal skills and competences; 
(c) job-related skills and competences. 


The transversal skills and competences can be split into six clusters as 
presented in Box 5. The foundations consist of three clusters. 


Box 5. ESCO transversal skills and competences and foundations clustering, 
as proposed in November 2019 


Transversal skills and competences 
A. Thinking skills 

e critical thinking 

e analysis and interpretation 

e creation and conceptualisation 

e reasoning/memory 
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entrepreneurship and performance 


People skills 

leadership and decision 
interaction and presentation 
conflict resolution 

support and cooperation 


Physical and manual skills 


manual skills for arts, crafts, music, etc. 


(Basic) learning skills 
numeracy 

literacy 

languages 


(Basic) life skills 
environmental literacy 
civic literacy 
economic literacy 
health literacy 


Foundations — personal traits — values-attitudes 


A. 


Intelligence 

creativity 

strategies 

intellectual processes 


Personality 
openness 
conscientiousness 
extraversion 
agreeableness 
emotional stability 


Interests 

artistic 

social entrepreneurial 
realistic 

conventional 


Source: Bjornavold et al., 2020. 
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The prototype development for the ERS took into account the ESCO skills 
pillar as available at the end of 2019. At the time of drafting this report (end of 
2020), the ESCO work on transversal skills and competences saw considerable 
progress. While it was not possible to consider this progress in the pre-test, it is 
important to present the further evolved thinking about the transversal skills. The 
most recent drafts on a structured and consistent terminology on transversal skills 
and competences suggest the following categories: 

(a) language skills and competences: the skills and competences referred to 
under this heading relate to technical mastery of a language with respect to 
the rules and conventions of language usage, such as order, meaning, 
grammar and expression; 

(b) thinking skills and competences: these allows the individual to deal with 
abstract cognitive concepts; 

(c) self-management skills and competences: self-management skills allow the 
individual to reflect on and make best use of his/her own abilities and potential; 

(d) social and communication skills and competences: these skills and 
competences allow the individual to interact with other people; 

(e) life skills and competences: these allow the individual to deal with the 
conditions and responsibilities of life in modern society. 


In this categorisation, physical and manual skills and competences are left 
out. Basic skills are not included as a separate category, as this would significantly 
overlap with, and to some extent confuse, the categories listed above. 


4.3.2. Use of existing national skills typologies 

The second approach was to use an existing reference point, as identified in actual 
ERS approaches. Of the studied examples (Australia, Austria, Ireland, Lithuania 1 
and 2, the Netherlands and Spain), the typology that seemed most adaptable was 
the one developed for the Australian ESS (the GAS-E). This skills list is 
manageable for graduates and employers and the terms are easy for graduates 
and employers to interpret; it is complete in terms of covering different types of 
learning outcomes (from transversal to occupation-specific ones); and finally, it is 
scalable and can be used for a variety of VET qualifications. The Australian list 
was developed by looking at a wide variety of frameworks that present skills lists 
related to graduate outcomes, curriculum outcomes, and workplace practice. The 
Australian qualifications framework took, for instance, inspiration from the four 
clusters of generic skills: 

(a) fundamental skills; 

(b) people skills; 

(c) thinking skills; 
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(d) personal skills. 


The preparatory analysis on existing frameworks (°°) concluded that, although 
different frameworks ‘are informed by very different motivations and philosophical 
underpinnings, there are similarities among the frameworks in terms of the skills 
(and in some cases attributes) that are identified’. (Oliver et al., 2014, p. 27). 
Another conclusion was that ‘one aspect that none of the frameworks covers well 
is technical skills and discipline-specific knowledge’ (Oliver et al., 2014, p. 27). The 
final version of the Australian list is firmly based on existing frameworks (in terms 
of more transversal skills) but with added technical skills and domain-specific 
knowledge. For the purpose of the prototype ERS, the Australian list can be used 
as inspiration, while needing some further reflection and finetuning. Below, the 
categories and respective items included in the Australian list are presented 
together with some further reflections and suggestions for improvement. 


Box 6. Suggestion of a reference point: the Australian list of skills and 
attributes accompanied with reflections 


A. Foundation skills 

a) oral communication skills 

b) written communication skills 

working with numbers 

ability to develop relevant knowledge 

ability to develop relevant skills 

ability to solve problems 

ability to integrate knowledge 

ability to think independently about problems 


ma 


> 
Se eS 


( 
( 
( 
( 
( 
( 
(g 
(h 
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Reflection: What could be added here is the ability to communicate in a foreign 
language. 


B. Adaptive skills and attributes 

a) broad background knowledge 

ability to develop innovative ideas 

ability to identify new opportunities 

ability to adapt knowledge to different contexts 
ability to apply skills in different contexts 

f) capacity to work independently 


Reflection: These skills and attributes actually point to the vertical dimension (how the 
complexity of learning increases from [one] level to another). Items (d) and (e) are 


(°°) See for an overview of the frameworks and classifications discussed see Appendix A 
and B in Oliver et al., 2014. 
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more fitting for EQF level 4 and higher compared to EQF level 2 and 3. The 
suggestion is to leave it as it is. 


C. Teamwork and interpersonal skills 

(a) working well in a team 

(6) getting on well with others in the workplace 

(c) working collaboratively with colleagues to complete tasks 

(d) understanding different points of view 

(e) ability to interact with co-workers from different or multicultural backgrounds 


Reflection: no further reflections. 


D. Technical and professional skills 

a) applying professional knowledge to job tasks 
using technology effectively 

applying technical skills in the workplace 
maintaining professional standards 
observing ethical standards 

f) using research skills to gather evidence 


Reflection: This set refers in broad terms to how the graduate is able to carry out 
occupation-specific work tasks without explicitly defining them in the list: it would 
mean something completely different in different occupations. There might be a slight 
overemphasis on technical skills, which would make it less fitting to less technical 
occupations (for instance in the healthcare sector). A suggestion could be to replace 
in (c) ‘technical skills’ with ‘occupation-specific skills’. Further, (f) might be more 
related to higher EQF levels (even above EQF level 4). Finally, specific occupational 
skills will be added per occupation to make the list better tailored to the occupation. 


E. Employability and enterprise skills 
) ability to work under pressure 
) capacity to be flexible in the workplace 
c) ability to meet deadlines 
) understanding the nature of your business or organisation 
) demonstrating leadership skills 
f) demonstrating management skills 
g) taking responsibility for personal professional development 
h) demonstrating initiative in the workplace 


Reflection: This set seems to be more appropriate to higher EQF levels. It is 
suggested to include other employability and enterprise skills, such as ability to follow 
instructions and to leave out items, such as ‘demonstrate leadership skills’ or 
‘demonstrate management skills’. Further, there seems to be an overlap with cluster 
B (i.e. work independently). 


Source: Social Research Centre, 2019, p. 93 (skills list); Cedefop (reflections). 
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4.3.3. Conceptual basis for the ERS skills reference point 

From the discussions on existing reference points, it becomes clear that the ERS 

skills reference point will have to balance between being as complete and 

comprehensive as possible (demanding a systematic approach to clustering and 
categorising skills) while being concise and understandable for employers and 
graduates to work with. 

For this purpose, the ERS designed in this study uses a specifically developed 
skills reference point that is based on the main lines of reasoning of the ESCO 
KSC lists and the reference point used in the Australian ESS. The development of 
the ESS reference point is based on the following starting point: each learning 
outcome that the VET provider, the graduate and the employer will reflect on can 
be described as a relationship between the holder of the learning outcome and a 
particular object (each learning outcome depicts a subject-object relationship): 

(a) the subject is the graduate him/herself. In order to operate the learning 
outcomes and maintain different relationships with different types of objects, 
the graduate needs to possess specific qualities (characteristics) that can be 
described in terms of foundations and attitudes. This can also include the 
place of the graduate in broader society (for example civic integration, 
environmental literacy); 

(b) the object is that what is outside of the subject and with what the subject 
maintains a specific relationship. Objects in this relationship can be other 
people, society at large, the work organisation, and occupation-related 
objects. It can also be the graduate him/herself, in terms of the subject 
reflecting on him/herself as object; 

(c) the relationship between holder (subject) and different objects can be 
described in terms of a power balance: 

e _ passive: the object exercises power over the subject; 

e neutral: subject and object work together in a power balance; 

e active: the subject exercises power of the object. 


While the ‘neutral’ allows to list what basis is needed for the relationship 
between subject and object to function, the ‘passive’ and ‘active’ allow to describe 
how the subject has to deal with the relationship in terms of compliance or 
changing the situation. This relationship can also be associated with the EQF 
column of the descriptors table, focusing on responsibility and autonomy (besides 
the knowledge and skills column). At the lower EQF levels, the autonomy and 
responsibility is limited, referring to a more passive power balance in which the 
subject is confronted with a rather well structured context and being instructed. At 
the higher EQF levels, the autonomy and responsibility is extended, referring to a 
subject that exercises power over objects, be it persons (supervise/instruct) or 
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work contexts (for instance to manage and transform work contexts). Hence, the 
relationship (power balance) provides an indication of the level at which the subject 
is expected to operate. 

In line with the distinction as proposed in the context of ESCO, the ESCO 
skills cluster ‘foundations — personal traits — values and attitudes’ is linked to how 
the subject deals with him/herself and deals with society; the ESCO ‘transversal 
skills and competences’ and the ‘job-related skills and competences’ relate to how 
the subject deals with other people, the work organisation and the occupation- 
related objects (°°). Table 13 brings the two dimensions of ‘object’ and ‘relationship’ 
together and indicates related learning outcomes. This table is used to check 
whether all possible combinations between relationships and objects are 
described. 

Based on this more theoretical exploration, the following skills reference point 
was proposed (Box 7). This reference point considers the theoretical background 
but aims at putting forward a concise reference point that employers and graduates 
can reflect on in a limited period of time. 


(©) When reflecting on the further developed ESCO conceptualisation by the end of 2020 
(Section 4.3.1), a similar orientation can be found, as used for the ERS skills reference 
point: to structure the skills from the inside (self) to the outside (society, broader value 
areas and life skills). 
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Table 13. Describing relationships and objects 
. Job-related skills and 
Transversal skills and competences 
(sfere)) competences 
(=<Tefe)) 
Type of en 
eppeet Graduate Work General Specific 
‘ organisation : occupation- occupation- 
tT f as CEO and work in Sooty related related 
Wiseay N general objects objects 
relationship 
Passive: 
the object 
exercises ¢ Work ethics 
[louse Mer 7 Oa ¢ Ethical 
the subject * Being instructed se ve P behaviour * Complying with «Dealing with 
« Self- * Serve others an 4 * Respect occupation- specific 
In terms of consciousness Attentive ae rules related (safety) procedures, 
EQF: (*) listening 5 Worliunder « Help others procedures processes and 
: i i tools 
reference to pressure in society (00 
lower EQF * Be flexible in 
levels the workplace 
(autonomy and 
responsibility) 
e¢ Teamwork : 
* Collaborative e Understanding 
Neutral: work the nature of 
4 i , , i Environment Maintainin 
subject and Health literacy Getting on with your business = «_ Env ¢ Maintaining 
aed work ¢ Numeracy : ae " or organisation al literacy occupation- ¢ Occupation- 
: 5 e Knowledge e Economic related specific 
together in a * Eileracy *: Respect er about the literacy knowledge and knowledge 
Powter + Languages * Communication organisation understanding 
balance skills and line of 
e Understanding business 
other viewpoints 
Active: ¢ Sense of 
the subject initiative/see 
exercises Opportunities/ 
power of the Adaptability; innovation 
object ability to change ° Work or ° Operating 
perspective « Instruct others independently * Developing new specific tools 
e Endurance e Leadership e Problem- 7 Contribute be cr Slag and changing 
In terms of ¢ Autonomy * Conflict handling solving mpreving i 2 4 d processes, 
EQF: ibili society nowiecge.any rocedures 
e Responsibility for e Management solutions pI 
reference to own actions ° Taking and tools 
higher EQF responsibility 
levels for personal 
(autonomy and professional 
responsibility) development 


NB: 


(*) The object of self-consciousness (the self) exercises power over subject (consciousness) in terms of making known 
itself. 


Source: Cedefop. 


(6) In the case of the graduate as object, there is a relationship between the graduate as 
subject (GS) and the graduate as object (GO). The relationship between GS and GO 
can again be passive, neutral or active. In the case of a passive relationship, GO is 
able to change the initial perspective of GS. In the case of a neutral relationship, GS 
and GO go hand in hand. In case of an active relationship, GS directs the GO. This 
GS-GO relationship relates to the concept of higher-order volitions as proposed in 
Frankfurt, 1971. 
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Box 7. Proposed reference point 


A. Personal characteristics and foundational skills as exercised in the 
workplace 


Cluster A refers to column 2 of Table 13 and includes skills related to passive use 
(e.g. self-consciousness); neutral use (e.g. foreign language skills); and active use 
(e.g. persistence and endurance). This cluster describes basic foundational skills and 
personal characteristics that a graduate could possess when entering a job: 

(a) oral communication skills; 

(6) written communication skills; 

(c) foreign language skills; 

(d) working with numbers; 

(e) self-reflection and critical thinking; 

(f) self-consciousness and taking responsibility for own actions; 
(g) adaptability and ability of change perspective; 

(h) persistence and endurance. 


B. Teamwork and interpersonal skills as exercised in the workplace 


Cluster B refers to column 3 of Table 13 and includes skills related to passive use 
(e.g. ability to follow instructions); neutral use (e.g. working well in teams); and active 
use (e.g. handling conflicts). This cluster describes how the graduate works in an 
interpersonal context: 


(a) working well in a team and working collaboratively with colleagues to complete 
tasks; 


(b) getting on well with others in the workplace and understanding different points of 
view; 

c) ability to interact with colleagues from different or multicultural backgrounds; 

d) ability to follow instructions; 

e) ability to instruct and/or lead others; 

f) ability to handle conflicts. 


Employability and enterprise skills as exercised in the workplace 


Cluster C refers to column 4 of Table 13 and includes skills related to passive use 
(e.g. work ethics, meaning complying with what is considered ethical in the 
workplace); neutral use (e.g. understanding the nature of the business); and active 
use (e.g. solving problems). This cluster describes how the graduate works in an 
organisational context and in the labour market: 


a) ability to work under pressure; 

b) comply with work ethics; 

c) reflect work processes and procedures; 

d) capacity to be flexible in the workplace; 

e) ability to meet deadlines; 

f) understanding the nature of your business or organisation; 
g) ability to manage processes/projects; 

h) taking responsibility for personal professional development; 
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(i) demonstrating initiative in the workplace and showing sense of initiative; 
(j) ability to solve problems. 
D. Societal values as exercised in the workplace (*) 


Cluster D refers to column 5 of Table 13 and includes skills related to passive use (to 
show ethical behaviour) and active use (reflect on own role in society). The skills 
related to the neutral use (economic/environmental literacy) are not included in this 
list for the ERS, as these are deemed challenging to reflect on for employers. This 
cluster describes how the graduate works in a broader societal context: 

(a) show ethical behaviour and comply with rules and regulations; 

(b) critical reflection on own role and place in society 


E. General occupation-related skills and competences as exercised in the 
workplace 

Cluster E refers to column 6 of Table 13 and includes skills related to passive use 

(e.g. observing ethical standards); neutral use (e.g. applying professional knowledge); 

and active use (e.g. using technology). This cluster describes some general 

occupation-specific skills and competences which are regarded as skills that can be 

generically (i.e. occupation-independently) described, while meaning something 

differently in different occupations (for instance, using technology effectively for an 

ICT service technician means working with diagnostic ICT equipment; for a health- 

care assistant, it refers to patients monitoring equipment): 

a) applying professional knowledge to job tasks; 

using technology effectively; 

applying technical skills in the workplace; 

maintaining professional standards; 

observing ethical standards; 

f) using research skills to gather evidence. 


F. Specific occupation-related skills and competences as exercised in the 
workplace 


Cluster F refers to column 7 of Table 13 and refers mainly to active use. These lists 
are specific for each occupation and these lists of 10-15 items are based on the 
learning outcomes mapping conducted in the first part of the overall study (WA1). The 
ESCO KSCs are selected that: 


(a) are included in many countries (at least eight out of the 10); 
(b) are not already covered in the full list (i.e. they are occupation-specific skills). 


This leads to the following two specific occupation-related lists: 
(a) For an ICT service technician: 

e provide technical documentation; 

e perform ICT troubleshooting; 

e use repair manuals; 

e configure ICT system; 

e administer ICT system; 

e maintain ICT server/system; 

e perform backups; 

e repair ICT devices; 
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e implement ICT recovery system; 
e manage ICT legacy implication; 
e use precision tools. 


(b) For a healthcare assistant: 

e monitor basic patient signs; 

e communicate with nursing staff; 

e empathise with the healthcare user; 
e interact with healthcare users; 

e provide basic support to patients; 

e identify abnormalities; 

e support nurses; 

e ensure safety of healthcare users; 
e convey medical routine information; 
e manage healthcare users' data; 

e conduct cleaning tasks. 


(*) These could also be integrated under personal characteristics. 
Source: Cedefop. 


4.3.4. Skills reference point for the ERS pre-test 

For presentational reasons, the order of the clusters was changed when presented 
to employers so that the questionnaires start with questions about the occupation- 
related skills and end with the personal characteristics. Further, clusters A, C and 
D are consolidated. Hence, while maintaining the theoretical orientation, the seven 
clusters as described in Section 4.3.3 are consolidated in four clusters in order not 
to overcomplicate the skills list to be used in a survey. The order is also changed, 
starting with the more occupation-related skills and competences and finishing with 
more transversal skills and competences. It was assumed that this list would be 
detailed enough to characterise VET programmes; it also includes specific 
occupation-related skills. At the same time, it is concise enough to be used for an 
employer survey and to allow cross-programme analysis. 

The final list of learning outcomes used in the pre-test is presented in Box 8. 
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Reference point used for the pre-test of the ERS 


A. General occupation-related skills and competences as exercised in the 
workplace 


This cluster describes some general occupation-specific skills and competences. 
A1: applying professional knowledge to job tasks 
A2: using technology effectively 
A3: applying technical skills in the workplace 
A4: maintaining professional standards 
A5: observing ethical standards 
A6: using research skills to gather evidence 


B. Specific occupation-related skills and competences as exercised in the 
workplace 


For an ICT service technician: 


ICT Bt: 
ICT B2: 
ICT B3: 
ICT B4: 
ICT B5: 
ICT B6: 
ICT B7: 
ICT B8: 
ICT BY: 


provide technical documentation 
perform ICT troubleshooting 

use repair manuals 

configure ICT system 
administer ICT system 

maintain ICT server/system 
perform backups 

repair ICT devices 

implement ICT recovery system 


ICT B10: manage ICT legacy implication 
ICT B11: use precision tools 


For a healthcare assistant: 


HC B1: 
HC B2: 
HC B3: 
HC B4: 
HC B5: 
HC B6: 
HC B7: 
HC B8: 
HC B9: 


monitor basic patient signs 
communicate with nursing staff 
empathise with the healthcare user 
interact with healthcare users 
provide basic support to patients 
identify abnormalities 

support nurses 

ensure safety of healthcare users 
convey medical routine information 


HC B10: manage healthcare users' data 
HC B11: conduct cleaning tasks 


C. Teamwork and interpersonal skills as exercised in the workplace 
This cluster describes how the graduate works in an interpersonal context. 
C1: working well in a team and working collaboratively with colleagues to complete 


tasks 
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C2: getting on well with others in the workplace and understanding different points 
of view 


C3: ability to interact with colleagues from different or multicultural backgrounds 
C4: ability to follow instructions 

C5: ability to instruct and/or lead others 

C6: ability to handle conflicts 


D. Employability and enterprise skills as exercised in the workplace 


This cluster describes how the graduate works in an organisational context and in the 
labour market. 


D1: ability to work under pressure 

D2: reflect work processes and procedures 

D3: capacity to be flexible in the workplace 

D4: ability to meet deadlines 

D5: understanding the nature of your business or organisation 

D6: ability to manage processes/projects 

D7: taking responsibility for personal professional development (keep up to date) 
D8: demonstrating initiative in the workplace and show sense of initiative 
D9: ability to solve problems 

D10: oral communication skills 

D11: written communication skills 

D12: foreign language skills 

D13: working with numbers 

D14: persistence and endurance 

D15: critically reflect on own role and place in society 


Source: Cedefop. 


4.4. Questionnaires for VET providers, graduates and 
employers/supervisors 


To avoid asking for information that is already available, the VET provider, in 
anonymised form, provided the following information on graduates to the 
researchers (Section 4.2, workflow step 2): 

(a) VET qualification/VET programme; 

(6) EQF level of the programme. 


The VET provider was also asked to provide, through a short questionnaire, 
a characterisation of the VET qualification, in terms of learning outcome 
orientation, as well as a rating of the importance of specific learning outcomes for 
entering a job. In this, the same items are addressed as in the graduate and 
employers’ questionnaire, so that any similarities and discrepancies can be 
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identified. In Tables 14, 15 and 16, three questionnaires are presented: for VET 
providers; for graduates; and for employers. 


Table 14. 


Introduction 


Characterisation of 
the VET 
qualification by the 
VET provider and 
indication of 
relevance 


Indication of the 
proficiency level 
provided by the VET 
programme 


Relevance of the 
learning outcomes 
provided by the VET 
qualification 


Source: Cedefop. 


Questionnaire for VET providers 


Answer category 


Please provide a short description of the VET qualification in terms of: 
(a) the education level at which the qualification is offered; 
(b) the (average) duration of the programme leading to the qualification in years or in months; 
(c) the extent to which learning at the workplace is included in the programme (including an 
indication of the share of workplace learning); 
the number of students currently enrolled in the programme leading to the qualification this 
school year; 
the number of graduates from the programme leading to the qualification in the last two 
school years. 


You are asked to provide a characterisation of the VET qualification and indicate the extent to 
which the VET programme provided specific skills to the graduate for effectively working in an 
enterprise. 
Skills are grouped into four clusters: 
e cluster A: general occupation-related skills and competences as exercised in the 
workplace 
e cluster B: specific occupation-related skills and competences as exercised in the 
workplace 
e cluster C: teamwork and interpersonal skills as exercised in the workplace 
e cluster D: employability and enterprise skills as exercised in the workplace 
Each cluster will be assessed separately. 


In relation to each skill, to what extent do you believe that the VET programme has provided 
them to the graduate for effective work in a company/organisation? 
If the skill is not relevant to the graduate's workplace, you can select ‘not applicable’. 
| believe the VET programme provided this skill (*): 
(a) not at all 
(b) to some extent 
(c) to a moderate extent 
(d) to alarge extent 
(e) to avery large extent 
(f) not applicable 


(*) Referring to the reference point (skills typology) 


For each of the skills mentioned under cluster A (general occupation-related skills and 
competences) and cluster B (specific occupation-related skills and competences), to what 
extent has the VET programme provided them to the graduate to make him/her ready for work? 


If the skill is not relevant to the graduate's workplace, you can select ‘not applicable’. 


The VET programme provided the skill at (*): 

(a) the basic level enabling work with intensive supervision, assistance and guidance of more 
experienced employee 
the sufficient level enabling autonomous execution of the work tasks with some supervision 
and quality control 
a high level enabling completely independent execution of tasks by assuming the 
responsibility of their quality control 
not applicable 


(*) Referring to the reference point (skills typology) 
The next set of questions asks about the skills you think are important for recent graduates to 
have when starting a related job. 


From the list of skills (*), please select the five most important and the five least important ones. 


(*) Referring to the reference point (skills typology) 
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Table 15. 


Screening 


Current 
employment 


Transition from 
education and 
training to 
employment 


Achieved 
learning 
outcomes 


Relevance of 
the content of 
the VET 
programme for 
the job 


Satisfaction 
with overall 
graduate 

preparation 


Questionnaire for VET graduates 


Answer category 


e What is your name? 
e What is your date of birth? (YYYY/MM/DD) 
e When did you graduate from the VET programme? (month and year) 
e What is your current employment status? (*) 
(a) employed in a job related to the VET qualification 
(b) employed in a job unrelated to the VET qualification 
(c) enrolled in a formal education/training programme 
(d) unemployed/inactive (e.g. not looking for a job) 


*) Routing: only those answering (a) will continue the survey 


e When did you start your current employment? 
e What is your job title? 
e Please describe your key work tasks in three sentences or less 
e Where do you work? 
(a) same city as my VET school 
(b) same region as my VET school 
(c) another region than where my VET school is located 
(d) another country than where my VET school is located 


Between graduation and starting your current job, what situation(s) apply? Choose one or more of 
the following situations: 
(a) | worked in another job (or jobs) related to the VET qualification 
(b) | worked in another job (or jobs) unrelated to the VET qualification 
(c) | enrolled in a formal education/training programme 
(d) | was unemployed and searching for a job 
(e) | was for any reason inactive (e.g. not looking for a job) 


In case of (b)-(e), how many months were you not in a job related to the VET qualification? 


In this section we are interested in your assessment of the extent to which the VET programme has 
prepared you for effective work in your company/organisation. 
The skills are grouped into four clusters: 
e cluster A: general occupation-related skills and competences as exercised in the workplace 
e cluster B: specific occupation-related skills and competences as exercised in the workplace 
e cluster C: teamwork and interpersonal skills as exercised in the workplace 
e cluster D: employability and enterprise skills as exercised in the workplace 
Each cluster will be assessed separately. 


To what extent do you believe that you have acquired the individual skills through the VET 
programme? 
If the skill is not required in the job you are doing, you can select ‘not applicable’. 
| believe | have acquired this skill (*): 
(a) not at all 
b) to some extent 
c) to amoderate extent 
d) to alarge extent 
e) to avery large extent 
not applicable 


= 


(*) Referring to the reference point (skills typology) 


The next set of questions concerns the skills that you think are important for a graduate to have when 


taking up a related job. Please answer these questions in relation to your current job. 
From the list of skills (*), please select the five most important and the five least important ones. 


(*) Referring to the reference point (skills typology) 


Overall, how well did the VET programme prepare you for your current job? 
(a) not at all 
(b) not well 
(c) well 
(d) very well 
(e) don’t know/unsure 
Please explain your answer. 
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Answer category 


May we contact you for further questions, for additional information/insights on the topics in this 
survey? 

(a) no 

(b) yes, you can contact me via email: 


Source: Cedefop. 


Table 16. 


Screening 
and 
confirmation 


Achieved 
skills 
(learning 
outcomes) 


Indication of 
the 
proficiency 
level 
provided by 
the VET 
programme 


Questionnaire for employers 


Is there someone working (or that has worked) in your company that recently (in the last two years) 
graduated from a VET college (in the remaining of the survey we refer to this person as 'the graduate')? 
(a) yes 
(b) no 


Is this VET qualification: 

) a formal requirement for entering the job (required by law) 

) a prerequisite for entering the job (required by the company) 

) adesired qualification for entering the job (preferable, nice to have) 
) not a preferred qualification 


What is your main occupation/position in the organisation? 
) company owner 

) manager 

) regular employee 

) mentor/in-company trainer 

HR staff 

other 


Stvaccuw 


What is the name of the company/organisation? 
How big is the company/organisation in terms of the number of employees? 
Please describe the main line(s) of business of the company/organisation (two to three sentences). 


What is the geographical focus of the company/organisation (multiple answers possible)? local 
(a) regional 
(b) national 
(c) European 
(d) international 


In this section we are interested in your assessment of the extent to which the VET programme has 
prepared the graduate for effective work in your company/organisation. 
Skills are grouped into four clusters: 
e cluster A: general occupation-related skills and competences as exercised in the workplace 
e cluster B: specific occupation-related skills and competences as exercised in the workplace 
e cluster C: teamwork and interpersonal skills as exercised in the workplace 
e cluster D: employability and enterprise skills as exercised in the workplace 
Each cluster will be assessed separately. 


In relation to each skill, to what extent do you believe that the VET programme has provided them to the 
graduate for effective work in your company/organisation? 
If the skill is not relevant to the graduate's current job, you can select ‘not applicable’. 
| believe the graduate acquired this skill: 
(a) not at all 
(b) to some extent 
(c) to a moderate extent 
(d) to alarge extent 
(e) to avery large extent 
(f) not applicable 


For each of the skills mentioned under cluster A (general occupation-related skills and competences) 
and cluster B (specific occupation-related skills and competences), to what extent has the VET 
programme provided skills to the graduate to make him/her ready for work? 

If the skill is not relevant in the graduate’s current job, you can select ‘not applicable’. 


The VET programme provided the skill at: 
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Relevance of 


(a) the basic level, enabling work with intensive supervision, assistance and guidance of more 
experienced employee 

(b) the sufficient level, enabling autonomous execution of the work tasks with some supervision 
and quality control 

(c) ahigh level, enabling completely independent execution of tasks by assuming responsibility for 
quality control 

(d) not applicable 


(*) Referring to the reference point (skills typology) 


The next set of questions asks about the skills you think are important for recent graduates to have 


ae oareung when starting a related job. 
ou cone From the list of skills (*), please select the five most important and the five least important ones. 
provided by 
the VET : (*) Referring to the reference point (skills typology) 
qualification 
Overall, how well did the VET programme prepare the graduate for their job? 
(a) not at all 
(b) not well 
(c) well 
(d) very well 
(e) don’t know/unsure 
Please explain your answer. 
Overall On the basis of your experience with the graduate you were reflecting on in this survey, how likely is it 


satisfaction 


that you would consider hiring another graduate of this VET programme/with this VET qualification if you 
had a corresponding vacancy? 
Would you say this is: 
(a) very unlikely to consider 
(b) unlikely to consider 
(c) neither unlikely nor likely to consider 
(d) likely to consider 
(e) very likely to consider 
(f) don’t know/unsure 
Please explain your answer. 


Consent to May we contact you for further questions, for additional information/insights on the topics in this survey? 
be (a) no 
approached (b) yes, you can contact me via email 


Source: Cedefop. 


4.5. Pre-test approach and results 

4.5.1. Implementation and responses 

Initially, the ERS applied a staged approach, where the VET provider sends a 
questionnaire to the VET graduate to be completed. In this questionnaire, the 
graduate is asked about the link between the VET programme and his/her 
occupation and whether the learning outcomes achieved are valuable for the job. 
It also includes a section on providing contact details of the employer/supervisor 
who may be asked to complete another questionnaire. It is emphasised that the 
data are anonymous: they are not used at individual level and are only presented 
in aggregated form. The employer/supervisor will also receive a questionnaire to 
reflect on the learning outcomes achieved by the VET graduate and whether these 
learning outcomes are relevant for the work in the enterprise. From the first test 
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experience (in February/March 2020), this approach did not seem to work properly, 
as only a few graduates agreed to provide employer contact details. The questions 
to provide contact details were probably too confrontational for the graduates and 
— despite clear statements to the contrary — the graduates may have feared that 
they would be assessed by their employers. Another reason could have been that 
graduates did not feel comfortable providing contact details without asking their 
supervisor first whether they would like to participate in the survey. A final aspect 
that hampered the approach was the COVID-19 crisis and the associated 
lockdowns, hampering overall willingness to participate in surveys. 

After a pause in the ERS pre-test due to the COVID-19 pandemic, it started 
up again in September/October 2020 with a slightly different approach. The staged 
approach was dropped and both graduates and employers were directly 
approached via the VET providers. The VET providers identified a number of 
employers who may have recently hired graduates. A question was added asking 
whether the employer recently hired a graduate from the participating VET 
provider. While still challenged by the COVID-19 situation and associated 
restrictions (and increasing workload, for instance, in the healthcare sector), this 
approach was more successful in reaching graduates and employers. The final 
data collection took place between September and November 2020. Table 17 
provides an overview of the total number of responses for each country, VET 
provider and qualification. 


Table 17. | Respondenis in the ERS pre-test 


Vilniaus paslaugy versio profesinio 
mokymo centras (Vilnius): 1 complete 1 complete 
healthcare assistant 


Karaliaus Mindaugo profesinio 
mokymo centras (Kaunas): 0 complete 0 complete 
LT healthcare assistant 


Alytaus profesinio mokymo centras 


(Alytus): healthcare assistant peniinlel Oromplcts 


Elektrény profesinio mokymo centras 
(Elektrénai): 0 complete 0 complete 
ICT technician 


Regional VET centre (ROC) Tilburg: 


healthcare assistant PGalnprels 12complets 
NL 
: . 2 complete: 2 complete: 
aul aenrnptag 1 at level 3 1 at level 3 
1 at level 4 1 at level 4 


NB: (*) completed for the qualifications at level 3 and level 4. 
Source: Cedefop. 
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4.5.2. Results for a single qualification (Netherlands ROC Tilburg: 
healthcare assistant (°)) 

The pre-test was conducted in the ROC van Tilburg school for care and wellbeing 
(School voor Zorg en Welizijn) and focused on the healthcare assistant qualification 
at level 3 (Verzorgende, I!G). The VET provider facilitated the distribution among 
graduates of the last two years (2018/19 and 2019/20) and approached a group of 
employers that regularly hire graduates (or that regularly have apprentices and 
interns from the VET provider). This resulted in 12 employer responses and seven 
responses from graduates. The employers ranged from organisations with 150 
employees to those with 5000 employees. They all have a regional focus, 
providing healthcare services. All employers indicated that the qualification is a 
formal requirement for entering the job. 


Characterisation of the VET programme by the VET provider 

Before discussing the assessment from the employers and graduates (closing the 
feedback loop), first the assessment — or better, characterisation — of the VET 
programme by the VET provider is presented. Figure 11 indicates the skills which 
the VET provider finds most and least important (column 2); the level at which the 
VET provider provided the occupation-specific skills (column 3) and whether the 
VET provider believes the graduates acquired the skills (column 4). 


(©) The pre-test only reached the envisaged responses for this VET provider/qualification 
to allow analyses of the data. 
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Characterisation of the healthcare assistant level 3 programme by the 
VET provider (Netherlands) 


A) CIES Tate Mare lcer-lielamey melalui ag 
(most and least important) 


4. Maintaining professional standards 


1. Provide technical documentation 


3. Use repair manuals 

4. Configure ICT system 

5. Administer ICT system 

6. Maintain ICT server/system 

7. Perform backups 

8. Repair ICT devices 

9. Implement ICT recovery system 
10. Manage ICT legacy implication 
11. Use precision tools 

1. Monitor basic patients’ signs 


2. Communicate with nursing staff 


4. Interact with healthcare users 
5. Provide basic support to patients 


6. Identify abnormalities 


8. Ensure safety of healthcare users 
9. Convey medical routine information 
10. Manage healthcare users' data 


11. Conduct cleaning tasks 


2. Getting on well with others in the workplace 
and understanding different points of view 


3. Ability to interact with co-workers from 
different or multicultural backgrounds 


4. Ability to follow instructions 


6. Ability to handle conflicts 


The VET programmes provided the 
skill at... 


the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the basic level enabling work with intensive supervision, 
assistance and guidance of more experienced employee 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 


Skills are recognised in the 
graduate (by VET provider) 
to a very large extent 


the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the basic level enabling work with intensive supervision, 
assistance and guidance of more experienced employee 


the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 

the sufficient level enablin. execution of the work 
tasks io! 

the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 


the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 


1. Ability to work under pressure 

2. Reflect work processes and procedures 
3. Capacity to be flexible in the workplace 
4. Ability to meet deadlines 


NA 


So eee 


6. Ability to manage processes/projects 


NA 


7. Taking responsibility for personal professional NA 


development (keep up to date) 


8. Demonstrating initiative in the workplace and 
show sense of initiative 


9. Ability to solve problems 


10. Oral communication skills 


Source: Cedefop (pre-test employer reflection survey 2020). 
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The characterisation shows that those skills that are considered least 
important are also those that the VET provider does not recognise in the graduates. 
This applies, for instance, to foreign language learning (D12) (which is not a 
compulsory part of the programme) or conducting cleaning tasks (B11) (which is 
not a specific part of the learning outcomes of the qualification). However, those 
skills that are a priority are not necessarily acquired to a very large extent. For 
instance, critically reflecting on the own role and place in society (D15) is a priority, 
but something that could be better developed and more strongly recognised in the 
graduates. 


Feedback loop: reflections from employers and graduates 

As a general overall assessment, employers and graduates are positive about 
whether the VET programme prepared the graduate for his or her job: 83% of 
employers and 86% of graduates indicated that the programme prepared the 
graduate well; 17% of the employers and 14% of the graduates even assessed the 
preparation as very well. Further, 58% of employers considered it very likely that, 
based on the experience with the graduate they were reflecting on, they would 
consider hiring another graduate of this VET programme. 

A more detailed question related to whether graduates have acquired the 
skills, and hence to what extent the respondents believe that the VET programme 
has provided the graduates with the skills for effectively working in a 
company/organisation. This question was asked of the VET provider (one 
respondent), the graduates themselves (seven respondents) and the employers 
(12 respondents). Figure 12 shows the average score per cluster of skills. 


Figure 12. Clusters of skills provided by the VET programme (Netherlands) 
VET school score =Graduatescore sEmployer score 


Cluster D: Employability and enterprise skills 


Cluster B: Specific occupation-related skills and 
competences 


Cluster A: General occupation-related skills and 
competences 


Cluster C: Teamwork and interpersonal skills Ee 


NB: N=20. 
Source: Cedefop (pre-test employer reflection survey 2020). 
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This cluster overview shows that employers and graduates are most positive 
in recognising specific occupation-related skills and competences and that this is 
also the case for the VET provider. Employers are of the opinion that the skills in 
the employability and enterprise skills cluster are the ones least offered. 

Figure 13 presents the same dat, but specified per individual skill. The skills 
are not grouped by cluster, but listed in order, from most to least recognised by 
employers. 

The VET provider and the employers and graduates differ most in their 
assessment of whether the following skills are acquired by the graduates: 

(a) employers and graduates more positive compared to the VET provider: 
e (C3: ability to interact with colleagues from different or multicultural 
backgrounds; 
e D4: ability to meet deadlines; 
e B11: conduct cleaning tasks; 
e D14: persistence and endurance; 
(b) employers more negative compared to the VET provider: 
e D2: reflect work processes and procedures; 
e B10: manage healthcare users' data; 
e D113: working with numbers; 
e A6: using research skills to gather evidence; 
e B9: convey medical routine information; 
(c) graduates more negative compared to the VET provider: 
e D2: reflect work processes and procedures; 
e A4: maintaining professional standards; 
e B7: support nurses. 


It is interesting that, compared to the VET provider, employers and graduates 
are generally more positive on employability and enterprise skills, such as meeting 
deadlines and persistence and endurance. This might not be a core competence 
offered by the VET programme but, in the labour market, these skills are well 
recognised. On the other hand, more data-related learning outcomes (manage 
data, work with numbers) are assessed as being less present compared to what 
the VET provider assumes. 

Besides asking whether the graduates actually have obtained the skills, the 
pre-test survey also asked about which learning outcomes are considered most 
and least important. Table 18 has the detail. 
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‘The VET programme has provided the graduates with the skills. | believe the graduate acquired this skill’ (Netherlands) 


Figure 13. 


= VET school score 


= Graduate score 


= Employer score 
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Table 18. Most and least important learning outcomes (Netherlands healthcare 
assistant) 
Most important Least important 
C1: working well in a team and working ? : ; 
collaboratively with colleagues to complete tasks Bie: foreign euguadaexts 
. ; ; : D6: ability to manage 
A1: applying professional knowledge to job tasks c 
‘ ae : processes/projects 
Employers D7: taking responsibility for personal professional ree : 
D4: ability to meet deadlines 
development (keep up to date) Bid condicroloaninatask 
B1: monitor basic patient signs DI 3: Sei on neta 
B5: provide basic support to patients 9 
D1: ability to work under pressure 
A1: applying professional knowledge to job tasks D12: foreign language skills 
Graduates C1: working well in a team and working B11: conduct cleaning tasks 
collaboratively with colleagues to complete tasks A3: applying technical skills in the 
D7: taking responsibility for personal professional workplace 
development (keep up to date) 
e pei Seas a cee LTE SEEN A2: using technology effectively 
B3: : Sarre a aie s a . B11: conduct cleaning tasks 
VET C 1: marca wellin 4 A ant ete D4: ability to meet deadlines 
provider a; : ; D5: understanding the nature of your 
esata oe cee > ne tasks business or organisation 
): critically reflect on own role and place in D12: foreign language skills 
society 
NB: N=20. 


Source: Cedefop (pre-test employer reflection survey 2020). 


In terms of most and least important learning outcomes, there are similarities 
in employer and VET provider responses, but there are also differences. They 
agree on working in teams (C1), applying professional knowledge (A1); they differ 
related to taking responsibility for professional development (D7) or empathising 
with healthcare users (B3) and observing ethical standards (A5). In terms of least 
important learning outcomes, employers and the VET provider share common 
ideas, but they also differ. Similarities concern foreign languages (D12), meeting 
deadlines (D4) and conducting cleaning tasks (B11). They differ with regard 
managing processes (D6), working with numbers (D13), and understanding the 
nature of the business (D5). 

Graduates and VET providers have similar and diverging views on the most 
and least important learning outcomes. For the most important learning outcomes, 
they agree on applying professional knowledge to job tasks (A1) and working well 
in ateam (C1). They differ on, for instance, the ability to work under pressure (D1); 
observing ethical standards (A5) and critically reflecting on own role and place in 
society (D15). 
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To conclude, while there is an overall high level of satisfaction among 
employers and the graduates concerning the learning outcomes delivered by the 
VET provider, there are learning outcomes that need further reflection and 
discussion between employers, graduates and the VET provider in terms of 
whether they are sufficiently acquired during the programme and/or related to their 
importance. Examples of learning outcomes that deserve discussion are: 

(a) B10: manage healthcare users’ data; 
(b) D2: reflect work processes and procedures. 


VET providers have a more positive view of whether graduates obtained these 
learning outcomes compared to employers and graduates. Further, the ability to 
work under pressure (D1) is considered a priority by graduates and is assessed as 
being acquired, but the VET provider is more negative about whether graduates 
obtained this learning outcome. Apparently, graduates feel that they have obtained 
more learning outcomes than the VET provider can oversee. 


4.5.3. Comparing qualifications: overall analysis results 

The data can also be used to provide an overall assessment of what skills are 
generally recognised in graduates. Based on the limited available data, some 
provisional analysis can be conducted to provide this overview. At this point, 
however, it is not possible to make comparisons between countries, to compare 
the same qualification profiles in the different countries, or to look at different 
qualification profiles within one country. Once more data becomes available, these 
possibilities could be tested. 


Characterisation of the VET programme by VET providers 

Before discussing the assessment from the employers and graduates (closing the 
feedback loop), first the assessment — or better, characterisation — of the VET 
programmes by the VET providers is presented. Figure 14 indicates the skills 
which the VET providers find most and least important (column 2); the level at 
which the VET provider provided the occupation-specific skills (column 3) and 
whether the VET providers believe the graduates acquired the skills (column 4). 
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Figure 14. Characterisation of all programmes together by the VET providers 
(Lithuania/Netherlands) 


Skills are recognised in the graduate 
Skills and indication of priority The VET programme provided the skill |(by VET provider) 
(most and least important) cee 1: not at all; 5: to a very large extent; 

(Fat ale) ar-1 0) )ITer=L0)(-) 


the sufficient level enabling autonomous execution of the work 
___ tasks with some supervision and quality control 
the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 
the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 
the basic level enabling work with intensive supervision, 
assistance and guidance of more experienced employee 
the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 
the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 
the basic level enabling work with intensive supervision, 
assistance and guidance of more experienced employee 


3. Applying technical skills in the workplace 


4. Maintaining professional standards 


competences 


Cluster A 
General occupation- 


6. Using research skills to gather evidence 


1. Monitor basic patients’ signs 


the sufficient level enabling autonomous execution of the work 


4. Interact with healtheare users tasks with some supervision and quality control 


5. Provide basic support to patients 


the basic level enabling work with intensive supervision, 


6. dently abnormalities assistance and guidance of more experienced employee 


competences 


Cluster B 


7. Support nurses 


the sufficient level enabling autonomous execution of the work 


8. Ensure safety of healthcare users tasks with some supervision and quality control 


9. Convey medical routine information 


Specific occupation-related skills and 


10. Manage healthcare users’ data 


the sufficient level enabling autonomous execution of the work 
tasks with some supervision and quality control 


2. Getting on well with others in the workplace and 
understanding different points of view 


3. Ability to interact with co-workers from different 
or multicultural backgrounds 


4. Ability to follow instructions 


Teamwork and 
interpersonal skills 


(oy [UE (=) a @ 


5. Ability to instruct and/or lead others 

6. Ability to handle conflicts 

1. Ability to work under pressure 

2. Reflect work processes and procedures 


3. Capacity to be flexible in the workplace 


6. Ability to manage processes/projects 

7. Taking responsibility for personal professional 
development (keep up to date) 

8. Demonstrating initiative in the workplace and 
show sense of initiative 


'y and enterprise skills 


9. Ability to solve problems 
10. Oral communication skills 


11. Written communication skills 


SEES ESS FSESESESESE ERE S 


13. Working with numbers 


14. Persistence and endurance 


- 


Source: Cedefop (pre-test employer reflection survey) 2020. 


The characterisation points to a higher priority (column 2) of general 
occupation-related skills and competences compared to employability skills (such 
as communication skills, foreign languages and working with numbers). A high 
level of skills provision (column 3) is mainly indicated for the healthcare assistant 
qualification occupation-specific skills. In terms of recognising skills (column 4), 


— 
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this is also more prominent in cluster A (general occupation-related skills) and 
cluster B (occupation-specific skills), compared to cluster C (interpersonal skills) 
and D (employability). 


Feedback loop: reflections from employers and graduates 

As a general overall assessment, employers and graduates are positive about 
whether the VET programmes prepared the graduates for their jobs: 79% of 
employers and 89% of graduates indicated that the programme prepared the 
graduate ‘well’; 21% of the employers and 11% of the graduates even assessed 
the preparation as ‘very well’. Further, 64% of employers regarded it as highly likely 
that they would consider hiring another graduate from the same VET programme 
based on the experience with the graduate they were reflecting on. 

A more detailed question related to whether graduates have acquired the skills 
and hence to what extent the respondents believe that the VET programmes have 
provided the graduates with the skills for effectively working in a 
company/organisation. This question was asked of the VET provider (seven 
respondents), the graduates themselves (10 respondents) and the employers (16 
respondents). Figure 15 shows the average score per cluster of skills. 


Figure 15. Clusters of skills provided by the VET programmes (both HC and ICT in 
Lithuania and the Netherlands) 


VETschool score sGraduatescore mEmployer score 


Cluster D: Employability and enterprise skills ————— 
Cluster C: Teamwork and interpersonal skills _— —___—____________™ 


HC cluster B: Specific occupation-related skills 


and competences —————————— errr 
ICT cluster B: Specific occupation-related skills 
and competences SSS ery 
Cluster A: General occupation-related skills and 
competences =a 
1 2 3 = 5 
Not at all ------------------------------------- Toa very large extent 


NB: N=33. 
Source: Cedefop (pre-test employer reflection survey 2020). 


The analysis shows that there is a large difference between the healthcare 
and ICT profiles concerning the specific occupation-related skills and 
competences. The employers and graduates (and also the VET providers) for the 
healthcare profile are more positive compared to the employers and graduates 
(and VET providers) for the ICT technician profile. 
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Figure 16 presents the same data, but then specified per individual skill. The 
skills are not grouped by cluster but listed in order from most to least recognised 
by employers. 

Besides the healthcare-specific skills (HC B5 and HC B3), employers 
recognise team work (C1) and following instructions (C4) most in the graduates. 
Flexibility (D3), applying professional knowledge (A1), maintaining professional 
standards (A4), working under pressure (D1) and persistence and endurance 
(D14) are also assessed as being present to a large extent. When comparing the 
healthcare assistant and ICT technician profile, it is noticeable that healthcare 
assistant skills are generally well recognised, while this is not the case for the ICT 
technician (see, for instance, ICT B8, B6, B9, B10, B11 all being at the lower end 
and hence not recognised by the employers in the graduate). This might suggest 
that the skills list for the ICT technician is not entirely matched to the qualifications 
assessed and the learning outcomes associated. An indication for this is that the 
occupation-specific skills in the list are quite technical in the sense that the ICT 
technicians are supposed to work with computer hardware and work, for instance, 
with precision tools. In reality, however, these technicians, as trained by the VET 
providers and employed in the companies, are supposed to work more with 
software settings. Hence these specific skills seem to be less relevant for the 
qualifications assessed in the pre-test. 

Besides asking whether the graduates actually obtained the skills, the pre-test 
survey also explored which learning outcomes are considered most and least 
important. When asked about this, the following picture emerges across the 
qualification profiles and countries (Table 19). 
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Table 19. Most and least important learning outcomes all qualification profiles 
and countries 
Most important Least important 
oy, eating Prefers Rrowoge 19 242 12: oan language sil 
Emplovers collaboratively with colleagues to complete tasks se aT : ey ae ree / 
ploy D1: ability to work under pressure ee y gep 
D7: taking responsibility for personal professional Z : : 
development (keep up to date) D13: working with numbers 
A1: applying professional knowledge to job tasks D12: foreign language skills 
D1: ability to work under pressure D13: working with numbers 
C1: working well in a team and working A3: applying technical skills in the 
Graduates F ; 
collaboratively with colleagues to complete tasks workplace 
D7: taking responsibility for personal professional A6: using research skills to gather 
development (keep up to date) evidence 
A1: applying professional knowledge to job tasks D5: understanding the nature of your 
VET C1: working well in a team and working business or organisation 
rouider collaboratively with colleagues to complete tasks D13: working with numbers 
P A3: applying technical skills in the workplace D15: critically reflect on own role and 
A5: observing ethical standards place in society 
NB: N=20. 


Source: Cedefop (pre-test employer reflection survey 2020). 


In terms of most and least important learning outcomes, there are similarities 
in the responses of the employers and VET providers, but there are also 
differences. They agree on working in teams (C1) and applying professional 
knowledge (A1); they differ on taking responsibility for professional development 
(D7) or applying technical skills in the workplace (A3) and observing ethical 
standards (A5). In terms of least important learning outcomes, employers and the 
VET provider share common ideas, but they also differ. Similarities concern 
working with numbers (D13). They differ with regard to managing processes (D6); 
critically reflecting on own role in society (D15) and understanding the nature of 
the business (D5). 

Graduates and VET providers have similar and diverging views on the most 
and least important learning outcomes. For the most important learning outcomes, 
they agree on applying professional knowledge to job tasks (A1) and working well 
inateam (C1). They differ on the ability to work under pressure (D1) and observing 
ethical standards (A5). 
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4.6. Emerging issues 


The following lessons can be drawn from the pre-test of the employer reflection 

survey: 

(a) the pre-test faced severe challenges in reaching out to employers and 
graduates. While the VET providers approached could be convinced to 
participate rather easily, it was more challenging than expected to have them 
reach out to employers and graduates. There are several reasons for this: 

e reaching out to graduates required VET providers to have a functioning 
alumni policy, in which contact details of graduates are kept up to date 
and that also assured that graduates provided consent in being 
approached for surveys after obtaining their VET qualification. For some 
of the VET providers, this information is not readily available: the pre-test 
had to rely on approaching all students that obtained a certain 
qualification, not knowing whether they are employed or pursuing further 
study. This led to a high number of respondents being excluded from the 
survey after the question on whether they are employed in a job related 
to the VET qualification; 

e the route from graduates to employers (asking graduates to provide 
contact details of their supervisors) proved to be a dead end in the pre- 
test. Apparently, graduates — despite clear explanations — felt that they 
would be subject to assessment by their supervisors. In the rerun of the 
pre-test, the approach was changed to contacting employers directly. For 
this purpose, the research team asked the VET provider to make an 
inventory of which employers could have hired graduates in the last two 
years. This did not pose a huge challenge in the healthcare sector where 
mainly larger organisations hire graduates regularly. In the ICT sector, 
however, this was more challenging as there are many smaller 
organisations that only occasionally hire graduates. Also, the sector 
seems to be more dynamic in terms of job changes; graduates change 
employers, making it more difficult to keep track of where graduates are 
employed. Finally, a high number of employers that responded to the 
survey invitation were excluded, as they did not recently hire graduates 
from the specific VET programme; 

(b) the questionnaires and the skills lists used in the pre-test seem to be 
appropriate. The skills lists are detailed enough to allow in-depth reflections 
on the content of the qualifications, but are short enough (38 items) to be used 
in a survey without burdening the respondents with tiresome lists of skills. This 
points to a good balance of complexity and simplicity of the reference point 
used. The average time for the employers to complete the full survey was 15 
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minutes. The structure with the four clusters worked well and allows a 
maximum of comparisons between different qualification profiles, while also 
allowing occupation-specific skills to be assessed. The occupation-specific list 
(cluster B) seemed to work better for the healthcare assistant compared to the 
ICT technician. The skills listed for the latter were less recognised by VET 
providers, graduates and employers; 

despite the lack of data, the ERS approach is an interesting and promising 
approach to closing the feedback loop, looking in detail at the content of 
qualifications. It allows more direct exchange between VET providers and the 
employers of graduates on whether what is envisaged in terms of learning 
outcomes is perceived as being present in the work environment; 

there is too little data for lessons to be learned about comparing the results 
for different qualification profiles and between countries. At the time of drafting 
this report, sufficient data were available only for one VET qualification offered 
by one VET provider, so no comparison across VET providers, qualification 
profiles and countries was possible. Also, it was not possible to benchmark 
one qualification against others. As data collection efforts still continue, the 
research team hopes to obtain additional data to permit such analyses. 
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5.1... Conclusions 


Based on the analysis so far, and supported by the piloting in Lithuania and the 
Netherlands, our main conclusions related to a strengthening of the feedback loop 
between the labour market and VET providers can be summarised as follows. 


Conclusion No 1 

There are valuable sources for the creation of skills intelligence, but existing 
methods and tools rarely relate to individual qualifications and usually do not 
consider actual learning outcomes as they are realised and perceived in the labour 
market. Moreover, they often focus exclusively on the demand side and do not 
consider the supply of education and training sufficiently. Thus, they are of limited 
use for completing the feedback loop. 

This study first explored what data is already available in the countries 
showing the relevance of qualifications for workers, employers and other labour 
market actors, and particularly the match between the intentions of the VET system 
and the needs of the labour market. To this end, we examined VET graduate 
tracking surveys, skill mismatch analyses and the European skills and jobs survey, 
online vacancy advertisements, including Cedefop’s skills-OVATE, forecasting 
procedures at national level and Cedefop's pan-European skills, as well as 
employer reflection surveys. Particular emphasis was placed on the extent to which 
these approaches are useful in completing the feedback loop based on learning 
outcomes: do they take into account actual or achieved learning outcomes as 
experienced and monitored by employers? 

The analysis of these methods and tools showed that they provide rich 
insights into the degree of match between skills supply and demand, into current 
skill demands and future trends, and that they all provide important data for the 
creation of skills intelligence. However, they are usually not sufficient to close the 
feedback loop, as they often refer to a higher aggregated level, rather than to 
individual qualifications, and generally do not relate to the perception of the 
learning outcomes achieved by graduates and realised on the labour market. VET 
graduate tracking surveys could be considered, at least in part, as approaches to 
closing the feedback loop but, in many cases, they focus on more general issues 
and only in a few on specific learning outcomes. They are concentrated on the 
graduates and do not consider the employers’ perspective or explore whether the 
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intended learning outcomes are actually achieved. Overall, a bias can be 
observed, as most of the approaches and tools have a strong or exclusive focus 
on skills demand on the labour market and do not consider skills supply sufficiently. 


Conclusion No 2 
Employer reflection surveys have the potential for completing the feedback loop 
based on learning outcomes but they are usually not conducted in a systematic 
way at national level. The examples identified show a variety of different 
approaches and there are several challenges associated with their design. Their 
usefulness in illustrating the match between the intentions of the VET system and 
the demands of the labour market seems to be assessed higher in countries with 
weaker governance and system links between the labour market and the VET 
system. 

Employer reflection surveys, defined as approaches in which employers (or 
their representatives) are asked to give their reflections on the relevance of 
qualifications in the labour market, could play an important role in completing the 
feedback loop, as they can examine whether employers are generally satisfied with 
the graduates and the learning outcomes they bring to the workplace. In the 10 
countries included in this study, we identified surveys that address employer 
perception of and demand for qualifications and employer reflection surveys 
without or with reference to learning outcomes included in qualifications. The latter 
could be seen as the most direct means of monitoring the link between intended 
and actual or realised learning outcomes. 

The study analysed and compared aspects of design and methodology of six 
examples identified by the national researchers in the European countries covered 
by this study, as well as one survey from Australia, in order to inform and inspire 
the development of a prototype to be tested. The characteristics identified include: 
(a) the surveys were most often initiated by government authorities, though they 

were generally conducted by a secondary partner. In two cases, a non- 

government authority was the initiator and the same organisation conducted 
the survey (university of applied science in one case and an employer 

association in the other). The surveys either focused on a regional or on a 

national level and targeted employers of graduates, usually without further 

specifying a demarcation for the target respondents; 

(b) the potential respondents were usually contacted directly, either by email or 
by telephone, and the surveys were conducted either digitally way or using 
(semi-structured) interviews; 

(c) in most cases, the questionnaire for the survey did not have a specific focus 
on learning outcomes, but most used concepts similar or related to learning 


121 


Review and renewal of qualifications: 
towards methodologies for analysing and comparing learning outcomes 


outcomes that are based on either an existing or specifically developed 
typology. Overall, the examples analysed showed a wide variety of ways in 
which these concepts were included and expressed in the questionnaires, 
how questions on learning outcomes were formulated and in what form the 
respondents’ answers were expected. In general, a bias towards transversal 
skills and competences can be observed in the skills lists used. 


While employer reflection surveys are actually often used for the review and 
renewal of qualifications, they are systematically conducted at a national level only 
to a limited extent. The research found that this is to partly due to the challenges 
associated with their design, that refer to the heterogeneous content of 
qualifications, the general difficulties related to the link between VET system and 
the labour market, and to the methodology of the surveys themselves. While in half 
of the countries covered by this study, the employer reflection surveys were 
considered as useful for gaining feedback from the labour market and improving 
the quality and relevance of VET qualifications, as well as for ensuring the 
engagement of employers, it was questioned whether such approaches would be 
useful and desirable in the other half. For example, in countries that already have 
in place strong links and functioning feedback mechanisms between labour market 
and the VET system, the introduction of regular employer reflection surveys might 
even be perceived by employers as an unnecessary administrative burden. In 
addition, in countries with dual systems (apprenticeships), specific approaches to 
collecting feedback from employers must be used, since employers are not only 
recipients of what the education and training system delivers and have a need for 
a particular skill set, but they are themselves involved in the supply process during 
in-company training. 

Based on the examples analysed and the exploration of the rationale behind 
the reluctance for conducting national employer reflection surveys, it can be 
assumed that such surveys provide the most direct added value for VET providers: 
If they are conducted at this level, they offer the VET providers an additional tool 
to engage with their graduates and the employers in their region, to initiate a 
dialogue with them, in order to inform the further development of their offers. The 
prototype developed for this study, therefore, focused on the provider level. 


Conclusion No 3 

The list of skills included in the reference point, as developed and used for the pre- 
test of the ERS, seems appropriate in so far as it allows for reasonable skills 
assessments, striking a balance between occupation-specific and generic skills, 
while keeping the total number of skills at a manageable level that allows their 
application in employer and graduate surveys. In addition, the balancing act of 
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developing a reference point — based on existing skills sets — while avoiding both 

oversimplification (with the risk of being potentially meaningless) and over- 

complexity (with the risk of not being understood by graduates and employers) 
seems to have been achieved. 

A first precondition for a working prototype is to have a reference point with a 
skill typology that is suitable for an employer survey. It should be able to achieve 
an appropriate level of detail in the description of learning outcomes, so that 
employers can reflect on them without presenting a list that is too long and too 
burdensome for employers to comment on. Taking into account international 
reference points and national examples of skills typologies, the research team 
designed a specific list that meets these requirements and also offers the 
possibility to apply it to different qualifications. The learning outcomes list consists 
of the following clusters: 

(a) general occupation-related skills and competences as exercised in the 
workplace: this cluster describes some general occupation-specific skills and 
competences (total six learning outcomes); 

(b) specific occupation-related skills and competences as exercised in the 
workplace: this cluster differs per qualification. In the pre-test for healthcare 
assistant and ICT technician, the list of 11 learning outcomes is based on the 
learning outcomes mapping conducted in WA1. The ESCO KSCs are selected 
that are included in many countries (at least eight out of the 10) and that are 
not already covered in the full list (i.e. they are occupation-specific skills); 

(c) teamwork and interpersonal skills as exercised in the workplace: this cluster 
describes how the graduate works in an interpersonal context (total six 
learning outcomes); 

(d) employability and enterprise skills as exercised in the workplace: this cluster 
describes how the graduate works in an organisational context and in the 
labour market (total 15 learning outcomes). 


In their feedback, employers indicated that the list was detailed enough to 
reflect on, especially when thinking about a specific graduate that is currently 
employed. This made it easier to think about whether the specific learning outcome 
is present for the graduate. 

While none of the existing skills lists seemed to be appropriate to be used for 
the prototype of an ERS, it is still important to build the ERS on existing reference 
points. Developing each ERS from scratch would probably undermine the use of 
the tool. Developing and pre-testing the ERS typology revealed lessons that are 
relevant for the work on the ESCO list of transversal skills and competences. This 
first relates to the underlying methodology for categorising skills. Both the ERS and 
the revised ESCO structure for transversal skills (state of play end of 2020) apply 
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an internal structure that is moving from skills related to the self to skills related to 
a wider context. From the pre-test, we learned that this leads to skills categories 
that are understandable and usable in the context of employer and graduate 
surveys. However, it is still challenging to separate clearly transversal and 
occupation-specific skills and competences. In the context of the ERS, the 
objective was to use a skills list that can be applied in different occupations and 
qualifications to allow comparison between them. Here the tendency is to identify, 
as much as possible, similarities and overlap in skills and competences and to limit 
the list of occupation-specific skills. A solution was to introduce general occupation- 
related skills and competences, which are transversal in the sense that they 
transcend particular skills (©), and so mean something different in different 
occupational contexts. These skills are not necessarily transferable between 
contexts. For ESCO, it could be relevant to explore further this tension between 
generic and job-specific competences and between transversal and transferable 
competences. While ESCO differentiates between four reusability levels of ESCO 
skills concepts (transversal, cross-sectoral, sector-specific and occupation-specific 
(§)), this does not thematise the transversality of skills (meaning something 
differently in different contexts) as described by Winch (2015). 


Conclusion No 4 

The pre-tested ERS approach is promising, but also challenging, as it requires VET 
providers to keep track of their graduates (alumni policy) and of the employers of 
their graduates. This is a challenge especially in sectors with many SMEs. 

The VET providers, including those for whom insufficient data was gathered 
from related employers and graduates, indicated that the approach is useful and 
evaluated the ERS instrument as positive. The feedback obtained this way was 
considered as important for the quality assurance of the VET programmes. 
However, they were much more concerned about the organisational and technical 
feasibility of surveys of graduates and employers. The ERS approach as used in 
the pre-test was — given the final response — challenging to implement. The main 
challenge was reaching out to graduates and their employers. In the first test (in 
February 2020), the idea was to approach the employers through the graduates 
(as implemented in the Australian ESS). This created too much of a bottleneck in 
the current set-up to reach any employers. The graduates did not feel comfortable 
providing employer contact details: this may have been because they considered 
this exercise as an individual assessment, or they did not feel comfortable checking 


(6) See Winch, 2015, p. 170. 
(4) https://ec.europa.eu/esco/portal/escopedia/Skill_reusability_level 
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with the employer whether it was ok to provide the contact details, or because the 
corona crisis reduced general willingness to participate in surveys. 

Reaching out to graduates and employers requires VET providers to have an 
active alumni policy. This was in place at the VET providers that acquired a good 
response rate but was not sufficiently in place at the other VET providers. This 
made lengthy the process of getting to a point where graduates and employers 
were addressed and failed to achieve high response rates. In addition, many 
employers who were approached appeared not to have recruited graduates from 
these VET providers in recent years. 

Establishing contact graduates and employers appeared to be more 
challenging in the ICT sector than in the healthcare sector. In the ICT sector, there 
is a less clear idea of who the employers of graduates are, as the sector consists 
of more SMEs. The labour market is also less regionally limited compared to the 
healthcare sector, and there appears to be a dynamic in the labour market that is 
related to the fact that graduates change jobs earlier at the beginning of their 
careers than healthcare graduates. 

Despite these challenges, the ERS pre-test led to interesting results (at least 
for one qualification) that can be used to support evidence-based dialogue 
between the VET provider and employers of graduates concerning the content of 
the qualifications. The employers consulted, after participating in the pre-test, 
agreed that it makes sense to have this dialogue between employers and VET 
providers in a direct manner and not just rely on providing input into the lengthy 
qualification renewal process about what employers would like to see in 
qualifications. The ERS offers a reality check for the VET providers on whether 
what they want to offer in the VET qualifications is also perceived — by the 
employers — as present in the graduates. 

The tested ERS approach asked both employers and graduates to reflect on 
the intended and actual learning outcomes. Closing the feedback loop could be 
done by only asking employers, but this would not provide the full story; the 
graduates have their own valid reflections on the VET qualification they have 
acquired and are rarely asked about the content of the qualification. They are 
consulted on their general satisfaction with the programmes and the organisation 
and their progression in the labour market (graduate tracking surveys), but usually 
not about whether they feel that they acquired the learning outcomes as envisaged 
by the VET provider. 
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Conclusion No 5 

The tested ERS approach is potentially scalable in terms of using the questionnaire 
for other qualification profiles, engaging other VET providers and offering different 
language versions. 

The ERS pre-test was conducted in Lithuania and the Netherlands and only 
covered the healthcare assistance and ICT technician profiles. The questionnaire 
was made available in English, Dutch and Lithuanian. As the questionnaires are 
short, and as the questionnaires for VET providers, employers and graduates 
overlap to a great extent, the effort needed to translate them into other languages 
is limited. Further, since three of the four parts of the skills list used are generic 
and only one part (around 11 learning outcomes) is specifically occupation-related, 
efforts to adapt the skills list to other qualification profiles can also be considered 
as manageable. It requires taking the ESCO profile, reducing the ESCO list by 
excluding all those skills that are covered by cluster A, C and D, and then selecting 
the most relevant 10 to 15 skills to be included in the skills list. In the pre-test, the 
research team used experiences from the first part of the overall study (work 
assignment 1) to select the skills that were assessed in different countries as the 
ones included in at least eight out of 10 national qualification descriptions (i.e. the 
core profile). 

Also, at a technical level, the ERS pre-test is scalable. The survey software 
allows different routings for different qualification profiles and can include different 
language versions. The generation of survey links for VET providers, employers 
and graduates was done manually and would require an automated approach 
when the ERS approach was to be scaled up. Lessons could be learned from the 
European Commission’s Joint Research Centre Selfie approach (°°), also currently 
being tested to be used by employers to reflect on the digital readiness of VET 
schools and associated learning environments. 

The main challenge, in terms of scalability, lies with VET providers and their 
infrastructure of reaching out to their graduates and employers of these graduates. 
It is not common practice that VET providers have an alumni policy or can easily 
reach out to employers. Also, capacities are lacking related to linking quality 
assurance and curriculum development issues to closing the feedback loops and 
considering reflections from employers and graduates. 


5.2. Recommendations 


The study leads to the following recommendations. 


(®) https://ec.europa.eu/education/schools-go-digital/now-selfie-works_en 
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Recommendation No 1 

It is recommended to integrate methods to complete the feedback loop more 
strongly into VET governance and quality assurance procedures and structures. 
This can also help to complement existing communication structures with 
evidence. The reflections from employers and graduates, and other labour market 
stakeholders more generally, provide valuable input into improving the content of 
the VET qualifications (i.e. the learning outcomes included). It also strengthens the 
involvement and engagement of labour market stakeholders with VET and their 
contribution to the design and content of VET qualifications. The reflection could 
not only focus on the intended and achieved learning outcomes but also consider 
how the curriculum is realised, through the methods of teaching and learning. 
Specific reflections are needed: to what extent labour market demand should be 
considered and VET should be employer driven; how the feedback from the labour 
market should be assessed and classified; what type of evidence should be 
considered; and which advice, and from whom, should be considered for the 
renewal of qualifications. Decisions on this are based on the respective values, 
beliefs and norms in a VET context and the ideologies expressed should be 
continuously reflected. The approach for integrating methods to complete the 
feedback loop more strongly into VET governance and quality assurance 
arrangements could be further explored and developed within the EQAVET 
framework in the context of the implementation of the VET recommendation 
(Council of the European Union, 2020). 


Recommendation No 2 

It is recommended to combine different sources for completing the feedback loop, 

as all sources have their specific added value and advantage. Together they shed 

light on: 

(a) the direct appreciation for specific skills on the labour market (vacancy 
analysis); 

(6) future demands and desires for skills on the labour market (skills forecasts); 

(c) how holders of qualifications (and their learning outcomes) navigate through 
the labour market and further education and training (VET graduate tracking 
initiatives); 

(d) how employers of graduates, the graduates themselves and the training 
providers assess the existing skills of graduates (employer reflection surveys 
as developed in this project). 


There are several ways to work on completing the feedback loop as discussed 
in this report. All have their specific approach and added value for gathering 
information to improve and tailor VET qualifications and to facilitate better links 
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between the VET sector and labour market stakeholders. Combining approaches 
can provide added value, since the more quantitative assessments (vacancy 
analysis and skills forecasts) are more valuable when they are enriched with more 
qualitative findings from VET graduate- and employer-related methodologies. As 
some approaches more strongly focus on the demand side, this bias can be 
avoided by combining them with approaches including the supply side. Also, 
sectoral skills bodies or professional associations, which may be responsible for 
or involved in the drafting of qualifications, could be more strongly involved in 
qualitative approaches, informing the review and renewal of qualifications. The 
results of a vacancy analysis focusing on the intensity of demand and the most 
frequently needed skills could be fed into an employer survey to validate the 
findings and to gain further insights into how the data can be interpreted. 
Specifically for the skills-OVATE approach, it could be recommended to explore 
the feasibility of including qualifications and their EQF levels in the data presented 
at skills-OVATE. 


Recommendation No 3 

It is recommended not to rely only on national or system-level feedback loops to 
gather information for the renewal of VET qualifications, but to strengthen more 
direct feedback loops between VET providers and their (local or regional) labour 
market stakeholders, so that reflections from the labour market can be taken on 
board in quality assurance and curriculum renewal more directly. Improvements in 
VET, also in terms of the content of qualifications, do not only rely on national or 
system level processes and how learning outcomes are written down in 
occupational profiles or other reference documents for qualifications. Within these 
processes and regulations, VET providers and their employers have the 
opportunity to adapt qualifications based on specific needs and considerations. 
Employer satisfaction surveys can initiate and contribute to the dialogue between 
VET providers and employers. 


Recommendation No 4 

It is recommended to experiment further with the employer reflection survey 
methodology and try to set up an infrastructure inspired by the Selfie-360 
methodology; this measures the digital readiness of schools by asking questions 
of school leaders, teachers, students and (in work-based learning settings) 
companies. The ERS, while being severely challenged, shows promising results 
and insights into the relationship between intended and achieved learning 
outcomes. Applying a Selfie-360 approach in gathering reflections on the achieved 
outcomes of VET programmes can: 

(a) stimulate discussions between VET providers and the labour market; 
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contribute to skills intelligence (when analysing data at sectoral/national level); 
strengthen involvement of labour market stakeholders in renewal processes 
and VET governance; 

contribute to international comparison and clarity about the outcomes of 
national VET qualifications. 


The following aspects need to be considered in the continued testing: 
expand to new countries. The ERS is pre-tested in Lithuania and the 
Netherlands, but it would be beneficial to obtain insights into whether the 
approach could also work in other VET contexts, such as those with a more 
dominant school-based VET system (as in eastern Europe countries) or with 
a more employer-based apprenticeship system (such as Denmark, Germany, 
Austria). Especially in the latter, it remains to be seen whether an ERS 
approach would at all add value (as expressed in Section 3.9); 

expand to more VET providers willing to participate and assure buy-in at 
national level and within economic sectors to test the approach. The pre-test 
only covered a small number of VET providers. More insight is needed into 
how the ERS approach works for a larger group of providers, operating in 
different contexts and with different organisational arrangements concerning 
employer engagement and alumni policies; 

expand to other qualification profiles besides the healthcare assistant and the 
ICT technician. The pre-test looked at two carefully selected occupations, but 
is limited in covering the full scope of VET qualifications. More insights are 
needed on how the ERS approach works in different occupational contexts. 
The same questionnaires can be applied (once translated and the specific 
occupation-related skills are identified and added for the new qualification 
profiles). Occupation-specific ESCO lists need to be analysed, overlaps with 
the ERS skills typology needs to be taken out, and a small set of specific 
occupation-related learning outcomes needs to be selected (based on some 
form of prioritisation); 

combine the testing of the ERS with support to VET providers to develop 
policies on employer engagement and alumni policies. The ERS approach 
primarily aims to support VET providers in their dialogue with employers and 
graduates on the content of VET programmes. The ERS approach can also 
be used to stimulate putting in place the necessary conditions for sustainably 
and operationally running the ERS, such as developing policies on employer 
engagement and alumni policies; 

combine the ERS outcomes at national or system level with information from 
other sources closing the feedback loop (vacancy analysis, skills forecasts) 
and discuss the combined findings with VET providers and employers. The 
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ERS is one approach among others: all have their specific added values. 
Obtaining the best overview of supply and demand of skills relies on 
combining the different sources. Further, to go beyond obtaining only written 
descriptions on supply and demand, the overview needs to be used to 
facilitate continuing dialogue between VET authorities, VET providers, labour 
market stakeholders (social partners), employers and graduates. 
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AAGE Australian Association of Graduate Employers (AU) 
AITSL Australian Institute for Teaching and School Leadership (AU) 
ANU Australian National University (AU) 
AQ Austria Austrian Quality Assurance body for Higher Education (AU) 
CATI computer-assisted telephone interviewing 
CBI Confederation of British Industry (UK-Eng) 
CBS Centraal Bureau voor de Statistiek 
Central Bureau of Statistics (NL) 
centre interprofessionnel de formation des commerces de |’ 
CIFCA alimentation 
interprofessional training centre for the food industry (FR) 
CNAE national classification of economic activities (ES) 
CNC computer numerical control 
CNCP catalogo nacional de cualificaciones profesionales 
national catalogue of professional qualifications (ES) 
CSO Central Statistics Office (IE) 
CVET continuing vocational education and training 
Direction de I’ Animation de la Recherche, des Etudes et des 
DARES Statistiques 
statistical service of the French Ministry of Labour (FR) 
DG EAC Directorate-General for Education, Youth, Sport and Culture 
DG EMPL pL Genet enol for Employment, Social Affairs and 
Inclusion 
directory of local units of economic activity 
DUAE 
(employer database, ES) 
EEA European Economic Area 
EQAVET URE al quality assurance in vocational education and 
training 
EQF European qualifications framework 
ERS employer reflection survey 
ESCO mpeuaied skills, competences, qualifications and 
occupations 
ESJ European skills and jobs 
ESS employer satisfaction survey 
FET further education and training (IE) 
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GAS-E graduate attributes scale —- employer (AU) 

GO graduate as object 

GOS graduate outcomes survey (AU) 

GS graduate as subject 

HC healthcare 

HE higher education 

HEA Higher Education Authority (IE) 

HR human resources 

ICT information and communications technology 

ISCED international standard classification of education 

IT information technology 

IVET initial vocational education and training 

KSC knowledge, skills and competences 

LEC Lithuanian Employers Confederation 

LM labour market 

LO learning outcomes 

MBO middelbaar beroepsonderwijs 
upper secondary vocational education (NL) 
nomenclature statistique des activités économiques dans la 
Communauté européenne 

NACE oon oe eas Ne bei 
statistical classification of economic activities in the 
European Community 

NAGCAS National Association of Graduate Careers Advisory Services 
(AU) 

NAVET National Agency for Vocational Education and Training (BG) 

NES national employer survey (IE) 

NLQF Netherlands qualifications framework 

NQF national qualifications framework 

OCW Ministry of Education, Culture and Science (NL) 
Office of Qualifications and Examinations Regulation 

Ofqual 
(UK-Eng) 

OJUV online job vacancy 

O*NET occupational information network (USA) 

PES public employment service 

QA quality assurance 
Stichting Samenwerking Beroepsonderwijs Bedrijfsleven 

SBB Foundation for Cooperation on Vocational Education, 


Training and Labour Market (NL) 
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SCQ skills, competences and qualifications 

Selfie self-reflection on effective learning by fostering the use of 
innovative educational technologies 

skills-OVATE skills online vacancy analysis tool for Europe 

SME small and medium-sized enterprise 

SOLAS Ant Seirbhis Oideachais Leanunaigh agus Scileanna 
Irish further education and skills service (IE) 

SSCs sector skills councils 

UAS University of Applied Science (AT) 

VET vocational education and training 

VQTS vocational qualifications transfer system 

WA work assignment 

WBL work-based learning 

WRC Workplace Research Centre (AU) 

WSSS worldskills standards specifications 


Country codes 


AT Austria 

AU Australia 
BG Bulgaria 

DK Denmark 
ES Spain 

Fl Finland 

FR France 

IE Ireland 

LT Lithuania 
NL Netherlands 
UK-Eng UK-England 
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ANNEX 1. 
Figure on learning outcome feedback loop and continuous dialogue 


Figure 17. Schematic overview of the feedback loops related to developing and renewing qualifications 
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Annex 1. 
Figure on learning outcome feedback loop and continuous dialogue 


NB: The yellow box in the middle schematically shows how learning outcomes descriptions ‘travel’ from the development of occupational standards and educational standards 
to their realisation by graduates being in the occupation related to the qualification. 
Purple boxes: Use of learning outcomes (LOs) in informing development, delivery and LM integration. 
Exchange/direct feedback (start feedback loop): The orange arrows indicate direct exchanges between VET providers and employers by using learning outcomes 
descriptions in their dialogue during the training provision, examination and certification processes and the labour market integration of graduates. These concern two-way 
communication channels: employers and VET providers also learn what might need to be changed in the learning outcomes descriptions as included in the qualification (or 
occupational standard, educational standard, curriculum). 
Bringing feedback to higher level (intermediate feedback loop): The green arrows show how information on the content of qualifications, the learning outcomes descriptions, 
is transferred from individual employers and VET providers to representatives and experts who are involved in the development and renewal of the qualifications and the 
learning outcomes descriptions. 
Capturing feedback in LOs (completing the feedback loop): The blue arrows indicate who is actually involved in the drafting of learning outcomes in the qualifications. Here 
the feedback loop, from work-practice back to the qualifications is completed. 
Support knowledge infrastructure: the grey circles and arrows show the general knowledge infrastructure on skills. This concerns graduate surveys, skills forecasting, 
reviews of the VET system, skills modelling, etc. 


Source: Auzinger et al., 2017, p. 123. 
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